As COVID-19 disrupted in-person survey operations, phone surveys proved a viable, useful, and cost-effective data collection mode that has since become widespread in low- and middle-income countries (LMICs). Phone surveys also respond to the need, in light of recurring shocks such as natural disasters, health epidemics, and violent conflict, for more rapid, frequent, and flexible data collection modes that can become part of routine data collection systems.
However, moving from traditional in-person data collection to mixed mode data collection including phone surveys creates new questions for methodological research. One such issue are survey mode effects, that is, differences in measured outcomes resulting from the data collection mode – in-person and over the phone. The existing evidence on mode effects proper is limited in the context of LMICs; but mode effects likely vary between different topics and question types, so there is a need for broad evidence on the matter.
In this study, we design a randomized survey experiment fielded as part of a large, representative household survey in Nigeria that comprehensively investigates survey mode effects across policy-relevant outcomes covering food security, health, labor, subjective welfare, migration, and other domains.
Intervention
We distributed phones to 937 households that answer identical questions in-person and over the phone. To isolate the effect of survey mode, we randomize the order of the in-person and phone interview and target the same respondents across interviews.
Experimental design
Our empirical strategy rests on randomly assigning respondents to answer identical questions in-person and over the phone across time, varying the order of the mode of interview.
As part of a separately funded methodological experiment on the mixed-mode measurement of agricultural labor inputs, the team distributed phones and sim cards to 995 households part of a large national panel survey in Nigeria (located in 106 randomly selected agricultural enumeration areas). The survey includes two in-person visits, one conducted in August/September 2023 and the second one conducted in February/March 2024. The proposed experiment will be fielded concurrently with the second in-person visit and involves scheduling phone interviews around the time of the in-person interview and administering a set of identical questions that can then be compared across survey modes.
We are interested in mode effects proper, that is, differences in measured outcomes due to differences in the survey mode. To isolate the effect of survey mode, our empirical strategy controls for the following confounders:
- Sample selection effects at the household level that would occur if households interviewed in-person were to differ from those interviewed over the phone - Respondent selection effects that would occur if respondents for the in-person survey were to differ from those of the phone survey - Anchoring/sequencing effects that would occur if respondents were to adjust their answers in whatever interview mode is conducted second to be consistent with the answers given in the first interview - Timing effects that would occur if both interview modes were conducted at different points in time such that recall or reference periods differed significantly
Design details
Randomization method
Randomization was conducted in Stata using the user-written program "randtreat" which is part of the ietoolkit package.
Randomization unit
We randomized at the household level, stratified by rural enumeration areas (106 EAs).
Clusters
937 households/respondents
Observations
937 households/respondents
Arms
471 households (Treatment Arm 1 - in-person first)
466 households (Treatment Arm 2 - phone first)
Primary outcomes
Health:
- Did respondent suffer from illness or injury?
- Did respondent consult a health practitioner or visited a health center?
- Total health expenditures (excl. insurance) by household in last 12 months
- Total health insurance expenditures by household in last 12 months
Subjective wellbeing:
- Cantril ladder questions: Self (today); Neighbors (today); Friends (today); Self (in 5 years)
- Self-assessed household income adequacy
- Locus of control: Life controlled by accidental happenings; Life determined by own actions; Life determined by others in household
- Likelihood of adverse effects of extreme weather events in next 12 months on household's finances
Migration aspirations and plans:
- Respondent would (still) like to leave community and migrate elsewhere
- Respondent (still) plans to leave community in next 12 months
Labor and non-farm enterprises:
- Any paid work outside the household in last 7 days
- Any work in own or household-member-owned non-farm household business in last 7 days
- Any agricultural labor in last 7 days
- Hours worked on a household farm or in other agricultural activity
- Any work in income-generating activity
- Any non-farm enterprise activity by any household member in last 12 months
Food consumption:
- Household food consumption score
- Household dietary diversity score
Shock exposure:
- Number of shocks experienced since January 2022
- Any shock experienced since January 2022
- Number of shocks with income and/or asset loss since January 2022
- Any shocks experienced since January 2022 that resulted in an income and/or asset loss
- Share of total shocks experienced since January 2022 that resulted in an income and/or asset loss
Secondary outcomes
Health:
- Any health expenditures (excl. insurance) in last 12 months
- Any health insurance expenditures in last 12 months
Migration aspirations and plans:
- Location where respondent considers moving (urban, rural, international)
Labor and non-farm enterprises:
- Intended use of agricultural products (own consumption, sale)
- Anyone in household owned or operated non-farm enterprise in last 12 months
- Anyone in household processed and purchased crops or livestock products for sale in last 12 months
- Anyone in household offered any service or hawking or sold anything on a street or market in last 12 months
- Anyone in household owned a professional office or offered professional services in last 12 months
- Anyone in household drove a household-owned or rented taxi in last 12 months
- Anyone in household owned a bar, restaurant, or food stand in last 12 months
- Anyone in household owned any other non-farm enterprise in last 12 months
Food consumption:
- Household dietary diversity score
- Food consumption by food category
Shock exposure:
- Extensive margin exposure to the most common among the following shocks: Floods; Irregular rains (including unexpected variation in timing and rainfall amount); Droughts; Fire; Very high temperatures (>40°C); Pests and Plant Diseases; Death of livestock due to illness; Post Harvest Loss; Death of a family member (including all causes); Diseases or injury of family member (including all causes); Stopped receiving remittances sent to the household; Loss of a regular job of a household member; Departure of a household member (Abandonment, separation, marriage); Loss of an important contract or default by a creditor; Nonfarm Business closure; failure or bankruptcy; Loss of land; Theft/looting of cash and other property; Hijacking/robbery/assault of a household member; Kidnapping/Abduction for ransom; Domestic violence; Withdraw of assistance (government, NGO, or other organizations); Increase in price of farming/business inputs (excluding petrol and other fuels); Fall in the price of farming/business output; Increase in price of major food items usually consumed by the household; Increase in price of oil and fuel;
Increase in prices of other fuels (excluding petrol) (e.g. cooking gas, kerosene, firewood, charcoal); Shortage/scarcity of petrol; Dwelling/farm buildings/business facilities damaged or demolished.
Cited by
No citation data available for this registration yet.