Reducing Nonresponse: Evidence from an Experimental Study
How to cite
Mohr, Aline. 2025. "Reducing Nonresponse: Evidence from an Experimental Study." AEA RCT Registry. AEARCTR-0017162. https://doi.org/10.1257/rct.17162-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Nonresponse in panel surveys reduces sample sizes and may hence decrease sample representativeness, particularly during the initial waves following recruitment. Maintaining participation immediately after recruitment is therefore crucial for ensuring long-term panel quality.
This experimental study investigates how personalized communication and monetary incentives affect nonresponse in a probability-based, bimonthly online panel. The experiment consists of the following three parts. In Experiment 1, newly recruited panelists are randomly assigned to one of four treatment groups: a printed “Thank-you” postcard, a handwritten “Thank-you” postcard, a printed “Thank-you” postcard with an unconditional €5 incentive, or a control group without intervention. Using data from the subsequent panel wave, we estimate an optimal treatment allocation via a policy tree that maximizes response rates under a fixed budget constraint. In Experiment 2, we evaluate the learned optimal policy by comparing its performance to random assignment. In Experiment 3, we replicate Experiment 2 with non-treated panel members from an earlier recruitment.
We explore the results in two separate papers. Paper 1 investigates the effect of personalized communication and monetary incentives on nonresponse, comparing the average treatment effects across the different treatment conditions. Furthermore, we analyze treatment effect heterogeneity across sociodemographic subgroups, Big Five personality traits and participants’ motivation to participate. Overall, we expect personalized communication and monetary incentives to reduce nonresponse. Moreover, we expect the printed “Thank-you” postcard with a monetary incentive to have the strongest effect, and the printed “Thank-you” postcard without a monetary incentive to have the smallest effect on nonresponse. Further, we expect the effectiveness of treatments to vary systematically across sociodemographic subgroups, Big Five personality traits, and participants’ motivation to participate. Specifically, we hypothesise that the printed “Thank-you” postcard with a monetary incentive has the strongest effect among participants who indicate monetary incentives as their primary motivation for survey participation. The handwritten “Thank-you” postcard is expected to have the strongest effect among those who indicate support for the research team as their primary motivation for survey participation. We expect all treatments to have the strongest effect on subgroups who are known to be at highest risk of nonresponse: individuals who have lower educational degrees and non-German participants.
In Paper 2, we explore whether optimal treatment allocation of personalized communication and monetary incentives in an adaptive survey design yields lower nonresponse than random assignment and whether this effect is generalizable to existing panelists.
Experimental design
The experiment consists of three parts and is conducted prior to the first and second panel waves of a probability-based, bimonthly online panel in 2026. The online panel studies individual attitudes and preferences relevant to political and economic decision-making processes.
For each survey wave, an email invitation is sent to all panelists on the first day of every odd month. The survey remains open for one month. Panelists receive an incentive based on the length of the questionnaire (about 20-25 minutes; €0.25 euros per minute), which is credited to their study account. In addition, participants receive a conditional yearly bonus of €10 if they complete all surveys, or €5 if they miss only one. Incentives can be cashed out, paid as an Amazon voucher, or donated to charity twice a year.
In addition to the standard recruitment interview questions (sociodemographic characteristics, Big Five personality traits, need for cognition), we ask participants about their motivations to participate (monetary incentive/ interest in study content/ desire to express opinion/ sample representativeness/ support for social science research/ support for the research team/ other). Participants first select all motivations that apply and then indicate their primary motivation among these. We plan to use these interview questions, and particularly, the motivation question to explore treatment effect heterogeneity.
The study follows a between-subject design with one factor comprising four levels.
Between September and November 2025, new panel members are recruited based on two different sampling frames (population register vs. commercial address provider). After recruitment, participants are invited to participate in their first panel wave starting on 1 November 2025.
In Experiment 1, we randomly assign the newly recruited panelists to one of four equally sized treatment groups within each stratum. Participants in the first treatment group T1.1 receive a “Thank-you” postcard with printed text, while those in the second group T1.2 receive the same postcard but with handwritten text. Participants in the third group T1.3 receive the “Thank-you” postcard with printed text along with an unconditional monetary incentive of €5. The fourth group T1.4 serves as a control group and therefore receives no treatment.
The “Thank-you” postcards for treatment groups T1.1 - T1.3. are sent on 15 December 2025 and their second panel wave opens on 1 January 2026.
After completion of the second panel wave in January, we use the resulting data to determine the optimal treatment allocation based on participants’ available characteristics. To do so, we implement a policy tree that maximizes survey response conditional on treatment-specific costs, subject to an overall budget constraint equal to the cost of random assignment across the four treatments.
In Experiment 2, we split the control group T1.4 into two equally sized subgroups in order to evaluate the learned optimal policy. In treatment group T2.1, we implement the optimal policy derived from wave 2. In treatment group T2.2, we repeat Experiment 1 and randomly assign participants to one of the four equally sized treatment groups. The treatment allocation in T2.1 and T2.2 is constrained to ensure equal overall costs. The “Thank-you” postcards are sent on 15 February 2026 and the third panel wave opens on 1 March 2026. Panel members in treatment groups T1.1 - T1.3 do not receive an additional intervention.
We use data from the third panel wave in March to evaluate whether the optimal treatment allocation yields higher response rates than random assignment.
For additional evidence on the efficiency of the optimal policy, we replicate Experiment 2 with non-treated panel members from an earlier recruitment in Experiment 3 (T3.1 and T3.2).
Design details
Cited by
No citation data available for this registration yet.