The effect of survey design and mode on completion rates and data quality for remote surveys in low-income settings.
How to cite
Owsley, Nicholas Calbraith. 2020. "The effect of survey design and mode on completion rates and data quality for remote surveys in low-income settings.." AEA RCT Registry. AEARCTR-0006001. https://doi.org/10.1257/rct.6001-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Remote surveys in low-income settings, particularly in East Africa, are common, though currently include a range of common practices without clear best practices and are predominantly the purview of market researchers, with limited application in academic research practices to-date. Yet, differences in survey response rates and data quality across different modes of remote data collection and survey design features is of key importance as remote research becomes more popular in light of COVID-19, both for academics and organizations. This study tests for differences in response rates, data quality and substantive responses across two of the most common modes of remote data collection - phone and SMS surveys - and across core features of the survey design - question order, pre-communication, and completion incentive. This study is conducted using a survey on COVID-19 in Kenya amongst a low-income urban population, and the substantive content covers economic, psychological, and physical effects of COVID and a set of basic economic preference outcomes. The study aims to give robust evidence to validate methods for researchers and provide best guidelines for optimal survey design.
Intervention
"Interventions" in this case include the range of survey design features which are randomly varied across administered survey invitations and surveys. Treatment assignment is to one variation of each of the 4 below treatment sets for survey design, according to blocked random assignment.
1. Mode
Participants are assigned to complete the survey using the following modes and platforms with a Phone: SMS assignment ratio at 1:3. Each list of participants is also randomly assigned across days and times so as to create the maximum possible overlap in implementation times across modes.
a. Phone survey
Survey conducted over phone by an enumerator using SurveyCTO as a data capture tool.
b. SMS
Survey conducted over SMS through the Telerivet SMS poll platform.
2. Incentives
Participants are assigned to receive one of the below incentive amounts, conditional on completion of the 4th module in the survey (approximately 60% of questions), in the ratio of 6:4:3 for 300Ksh: 150 Ksh: 50 Ksh. In each of the below treatments, the amount is communicated in the initial script in the same way across modes during consenting.
a. Micro-incentive/reimbursement: Ksh 50 incentive
b. Reasonable incentive for length of survey: Ksh 150
c. Higher than average incentive for length of survey: Ksh 300
3. Survey Pre-communication
Each participant is randomly assigned to receive the following pre-communication regarding the survey in equal proportions. Please see here for a more detailed protocol.
a. No SMS
Respondents do not receive an SMS. The survey itself is the first communication they receive.
b. Plain SMS warning about the survey.
Respondents receive an SMS on the morning of or morning prior to the survey to indicate that the respondent will be invited to participate in a survey. The script is as follows:
Order Effects
The survey includes 2 modules which vary in position across the surveys 7 modules,, each with 1) a likert question regarding COVID sentiment, 2) a continuous response preference measure, and 3) a data quality measure. These modules are numbered 2 and 6. Each will be randomly assigned to be either early in the survey or late in the survey, as shown below:
a. Order 1
Module 2 → 6: The order of the survey will be -
Module 1
Module 2
Module 3, 4, 5
Module 6
Module 7
b. Order 2
Module 6 → 2: The order of the survey will be -
Module 1
Module 6
Module 3, 4, 5
Module 2
Module 7
Experimental design
Each participant is invited to complete a survey on COVID-19 via a respective remote survey mode and according to the randomly assigned features described in treatment conditions below. The survey asks questions regarding COVID-19 in Kenya amongst a low-income urban population, and the substantive content covers economic, psychological, and physical effects of COVID and a set of basic economic preference outcomes. This trial is primarily concerned with meta-outcomes of the survey rather than substantive outcomes.
The questions and script for the survey are the same across treatment conditions, but small features of the survey design are varied to identify treatment effects. The following sets of treatments are randomly assigned in a cross-cutting/blocked manner such that each survey invite will include a specific Mode (a or b, described in detail below), Incentive (a, b, or c), Survey pre-communication (a or b), and Order (a or b) according to assignment. Randomization will be blocked by treatment to maximize power for identifying primary effects and for interaction effects. In some cases, treatment blocks are deliberately unequal to account for budgetary constraints and low projected completion rates, but in such cases the sample is expanded to compensate for losses in power.
Outcomes evaluated include meta-survey outcomes, such as consent and survey completion (these are analysed across the set of treatments 1-3 outlined below) and also an analysis of data quality and of substantive outcomes but as they vary according to survey design features (according to treatment sets 1 and 4 below).
Treatment sets are as follows.
1. Mode
Participants are assigned to complete the survey using the following modes and platforms with a Phone: SMS assignment ratio at 1:3. Each list of participants is also randomly assigned across days and times so as to create the maximum possible overlap in implementation times across modes.
a. Phone survey
Survey conducted over phone by an enumerator using SurveyCTO as a data capture tool.
b. SMS
Survey conducted over SMS through the Telerivet SMS poll platform.
2. Incentives
Participants are assigned to receive one of the below incentive amounts, conditional on completion of the 4th module in the survey (approximately 60% of questions), in the ratio of 6:4:3 for 300Ksh: 150 Ksh: 50 Ksh. In each of the below treatments, the amount is communicated in the initial script in the same way across modes during consenting.
a. Micro-incentive/reimbursement: Ksh 50 incentive
b. Reasonable incentive for length of survey: Ksh 150
c. Higher than average incentive for length of survey: Ksh 300
3. Survey Pre-communication
Each participant is randomly assigned to receive the following pre-communication regarding the survey in equal proportions. Please see here for a more detailed protocol.
a. No SMS
Respondents do not receive an SMS. The survey itself is the first communication they receive.
b. Plain SMS warning about the survey.
Respondents receive an SMS on the morning of or morning prior to the survey to indicate that the respondent will be invited to participate in a survey. The script is as follows:
4. Order Effects
The survey includes 2 modules which vary in position across the surveys 7 modules,, each with 1) a likert question regarding COVID sentiment, 2) a continuous response preference measure, and 3) a data quality measure. These modules are numbered 2 and 6. Each will be randomly assigned to be either early in the survey or late in the survey, as shown below:
a. Order 1
Module 2 → 6: The order of the survey will be -
Module 1
Module 2
Module 3, 4, 5
Module 6
Module 7
b. Order 2
Module 6 → 2: The order of the survey will be -
Module 1
Module 6
Module 3, 4, 5
Module 2
Module 7
Design details
Cited by
No citation data available for this registration yet.