Quantifying and Addressing Coverage Bias in Phone Surveys

How to cite

Collins, Elliott; Glazerman, Steve. 2021. "Quantifying and Addressing Coverage Bias in Phone Surveys." AEA RCT Registry. AEARCTR-0007608. https://doi.org/10.1257/rct.7608-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Elliott Collins* , Steve Glazerman

* Primary investigator

Registry ID
AEARCTR-0007608
DOI
https://doi.org/10.1257/rct.7608-1.0
First registered
Last updated
Status
Completed
Trial dates
2021-01-28 – 2021-03-31
Countries
Philippines ()
JEL codes
C83
Registry record
http://www.socialscienceregistry.org/trials/7608

Abstract

With pandemic conditions making face-to-face data collection too risky or costly for most countries, global poverty researchers must rely heavily on mobile phone surveys. The main weakness of phone surveys is the difficulty achieving a representative sample (Gibson et al. 2017). Coverage bias will typically be correlated with outcomes of interest, introducing bias into most analyses of the survey data if not accounted for. This study takes advantage of a useful opportunity to test methods for addressing this sort of coverage bias in phone surveys, in which researchers are conducting a phone survey of a large sample of enrollees in a digital cash transfer program managed by a government agency. The researchers have baseline data for each potential respondent, including location, household demographics, and economic outcomes of interest from the original cash transfer program. Potential respondents are randomly placed into either high-intensity or low-intensity contact protocols, similar to the approach recommended in DiNardo et al. (2021), generating random variation in the probability that a respondent will end up in the sample, and allowing researchers to assess the marginal value of each follow-up call, given they did not answer on the previous attempt. Using the economic features in the baseline data as outcomes, we can then assess the effectiveness of various weighting schemes designed to adjust for coverage bias.

Intervention

Prior to starting phone calls, the experiment separates the call list of cash transfer recipient phone numbers (N=5000) evenly into three groups:
- Control Group numbers receive one call in each of three time blocks over a single day, after which they are replaced.
- SMS Group numbers follow the control protocol, but with an SMS message sent in advance of the second call explaining who is calling and why.
- High Intensity Group numbers follow the same protocol as the SMS Group, but over two days before being replaced, rather than one.

Another 1,500 phone numbers were selected as replacement numbers.

Treatment status was stratified by Region, Financial Service Provider (FSP) used by the recipient, and the date of the initial cash transfer, and replacement is conducted entirely within-strata.

Experimental design

Prior to starting phone calls, the experiment separates the call list of cash transfer recipient phone numbers (N=5000) evenly into one of the three treatment groups. Another 1,500 phone numbers are selected as replacement numbers. Treatment status was stratified by Region, Financial Service Provider (FSP) used by the recipient, and the date of the initial cash transfer, and replacement is conducted entirely within-strata.

Researchers will first characterize the differences in demographic and economic profile between the full sample (based on baseline data) and the phone sample (those who ultimately answered the phone and completed a survey). The analysis will then estimate the marginal response rate for each additional call made to unresponsive numbers, with response rates calculated according to AAPOR guidelines, providing an estimate of the value of recruitment intensity for increasing sample size in phone surveys. The demographic differences observed among those contacted on the first, second, and third attempts will be used to characterize the value of recruitment intensity for addressing coverage bias.

Design details

Randomization method
Randomization was done using a random number generator by the same team that managed data collection and cleaning.
Randomization unit
Randomization was conducted at the individual level within strata.
Clusters
5000 phone survey respondents.
Observations
5000 phone survey respondents.
Arms
Each sample contains 1666 or 1667 phone numbers.
Primary outcomes
1. Survey Consents and Completions 2. Average age, gender, location, and economic profile of respondents

Cited by

No citation data available for this registration yet.

Keywords