How to cold call firms? An application of multi-armed bandit optimization in corporate web surveys
How to cite
Rostam-Afschar, Davud; Gaul, Johannes; Simon, Thomas. 2022. "How to cold call firms? An application of multi-armed bandit optimization in corporate web surveys." AEA RCT Registry. AEARCTR-0009801. https://doi.org/10.1257/rct.9801-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Previous experiments show that (even slight) differences in the design of invitation letters result in significant differences of starting rates between the respective groups. Typically, these studies rely on static experimental groups and inference that allows to determine the best performing invitation letter only ex-post. Trading off exploration and exploitation, we apply a reinforcement learning method in an adaptive experiment, as proposed by Scott (2010). Using multi-armed bandit optimization, specifically Thompson Sampling, we may compare a substantially larger set of alternatives. We show how Thomson Sampling can be implemented in the context of survey research conducted by the German Business Panel (GBP) and how accurate inference can be drawn. We aim to identify the distinct elements of a corporate web survey invitation letter that maximize survey starting rates.
Intervention
We conduct an adaptive experiment using Thompson Sampling instead of an experimental setup with static group composition. The main advantage of using a reinforcement learning algorithm is that insights about treatment effects are continuously exploited while the experiment is conducted, optimizing output (survey starting rates) already during the exploitation phase of the (sequential) experiment. In addition, such research design allows us to compare a larger number of treatments than in previously conducted experiments that consider a maximum of 16 different invitation letters (Kaplowitz et al., 2011).
Our research design varies five key elements of a survey invitation letter. Specifically, we consider (1) the personalization of invitation letters, (2) emphasizing the authority and status of the sender, (3) survey link placement, (4) stressing compliance with data protection, and (5) using a distinct request style by offering potential survey respondents the opportunity to share their opinion or pleading for help. We implement each of these factors in two different ways, resulting in a full-factorial experiment with a total of 32 (2^5) treatments.
Experimental design
Randomization is conducted via random allocation of subjects into 32 different treatment groups. Yet, group sizes might differ during the course of the experiment. Within the first four weeks of the survey period, allocation to all groups happens with the same frequency.
Following this burn-in-phase, each week the bandit algorithm evaluates realized treatment effects and dynamically determines group sizes. In doing so, the bandit algorithm prioritizes relatively strong treatments versus relatively weak ones, while still testing the latter. Weights for allocation (size of the treatment groups) are calculated according to the formula stated in the section “Sampling design” of the research proposal. In sum, subjects are allocated to treatment groups at random, however the probability of receiving a specific treatment is updated weekly and is equivalent to the weights calculated by the Thompson Sampling algorithm.
Design details
Cited by
No citation data available for this registration yet.