Quantifying and Addressing Coverage Bias in Phone Surveys
How to cite
Collins, Elliott; Glazerman, Steve. 2021. "Quantifying and Addressing Coverage Bias in Phone Surveys." AEA RCT Registry. AEARCTR-0007608. https://doi.org/10.1257/rct.7608-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
With pandemic conditions making face-to-face data collection too risky or costly for most countries, global poverty researchers must rely heavily on mobile phone surveys. The main weakness of phone surveys is the difficulty achieving a representative sample (Gibson et al. 2017). Coverage bias will typically be correlated with outcomes of interest, introducing bias into most analyses of the survey data if not accounted for. This study takes advantage of a useful opportunity to test methods for addressing this sort of coverage bias in phone surveys, in which researchers are conducting a phone survey of a large sample of enrollees in a digital cash transfer program managed by a government agency. The researchers have baseline data for each potential respondent, including location, household demographics, and economic outcomes of interest from the original cash transfer program. Potential respondents are randomly placed into either high-intensity or low-intensity contact protocols, similar to the approach recommended in DiNardo et al. (2021), generating random variation in the probability that a respondent will end up in the sample, and allowing researchers to assess the marginal value of each follow-up call, given they did not answer on the previous attempt. Using the economic features in the baseline data as outcomes, we can then assess the effectiveness of various weighting schemes designed to adjust for coverage bias.
Intervention
Prior to starting phone calls, the experiment separates the call list of cash transfer recipient phone numbers (N=5000) evenly into three groups:
- Control Group numbers receive one call in each of three time blocks over a single day, after which they are replaced.
- SMS Group numbers follow the control protocol, but with an SMS message sent in advance of the second call explaining who is calling and why.
- High Intensity Group numbers follow the same protocol as the SMS Group, but over two days before being replaced, rather than one.
Another 1,500 phone numbers were selected as replacement numbers.
Treatment status was stratified by Region, Financial Service Provider (FSP) used by the recipient, and the date of the initial cash transfer, and replacement is conducted entirely within-strata.
Experimental design
Prior to starting phone calls, the experiment separates the call list of cash transfer recipient phone numbers (N=5000) evenly into one of the three treatment groups. Another 1,500 phone numbers are selected as replacement numbers. Treatment status was stratified by Region, Financial Service Provider (FSP) used by the recipient, and the date of the initial cash transfer, and replacement is conducted entirely within-strata.
Researchers will first characterize the differences in demographic and economic profile between the full sample (based on baseline data) and the phone sample (those who ultimately answered the phone and completed a survey). The analysis will then estimate the marginal response rate for each additional call made to unresponsive numbers, with response rates calculated according to AAPOR guidelines, providing an estimate of the value of recruitment intensity for increasing sample size in phone surveys. The demographic differences observed among those contacted on the first, second, and third attempts will be used to characterize the value of recruitment intensity for addressing coverage bias.
Design details
Cited by
No citation data available for this registration yet.