Who Guards the Data? Enumerator Effort, Honesty, and Survey Quality?

How to cite

Nigus, Halefom Yigzaw; Meles, Tensay Hadush; Abay, Kirom; Abate, Gashaw T; Chamberlin, Jordan; GEBREKIDAN, Bisrat Haile. 2026. "Who Guards the Data? Enumerator Effort, Honesty, and Survey Quality?." AEA RCT Registry. AEARCTR-0018747. https://doi.org/10.1257/rct.18747-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Halefom Yigzaw Nigus* , Tensay Hadush Meles , Kirom Abay , Gashaw T Abate , Jordan Chamberlin , Bisrat Haile GEBREKIDAN

* Primary investigator

Registry ID
AEARCTR-0018747
DOI
https://doi.org/10.1257/rct.18747-1.0
First registered
Last updated
Status
In development
Trial dates
2026-05-24 – 2026-09-30
Countries
Private
JEL codes
C80; C93; D91
Registry record
http://www.socialscienceregistry.org/trials/18747

Abstract

Household surveys are central to empirical research, yet little is known about how professional survey enumerators---frontline agents in data production---shape interview administration, data quality, and content of survey responses. Using randomized assignment of enumerators to respondents, we test whether survey outcomes differ across respondents assigned to enumerators with different pre-fieldwork incentivized measures of effort, honesty, and views on politics and governance. Before fieldwork, we elicit enumerators’ effort and honesty using incentivized tasks: a real-effort task that measures diligence in a routine but demanding exercise, and a private-reporting task that measures willingness to misreport private information for monetary gain. We also elicit enumerators’ cognitive and socio-emotional skills, as well as their own answers to political and governance questions included in the respondent survey. We then link enumerators to the interviews they conduct in large surveys conducted in Ethiopia and Kenya. The preregistered analysis tests whether enumerator effort, honesty, and their skills and perceptions affect survey data and survey responses. By combining incentivized measures with randomized field assignment, the study provides evidence on how traits, skills, and views of enumerators shape survey data, with implications for recruitment, training, and supervision in large-scale field studies.

Intervention

The study is being embedded in forthcoming large-scale household surveys in Ethiopia and Kenya. The intervention involves random assignment of enumerators to respondents as well as a lab-in-the-field experiment that entails eliciting honesty and effort levels. The scale and structure of the surveys, which allow random assignment of enumerators across respondents, provide a unique opportunity to measure the effect of enumerators’ important behavioral attributes (including honesty and effort) on survey data quality.

Intervention One: Random Assignment of Enumerators to respondents
The key intervention of this study is the random assignment of enumerators to respondents. This follows recent studies aiming to study the effect of enumerators (Di Maio and Fiala, 2020; Kerwin and Ordaz Reynoso, 2021; Rodriguez-Segura and Schueler, 2023; Kerwin and
Ordaz Reynoso, 2021; Kadam et al., 2025;West and Blom, 2017). We do this random assignment in two stages: First, we randomly assign enumerators to enumeration areas (villages). Second, within each enumeration area (village), we randomly assign respondents to those enumerators assigned to the corresponding enumeration area.

Intervention Two: Measuring survey enumerators’ honesty and effort levels as well as personality traits, cognitive skills, sociomotional skills and political opinions.

Beyond the random assignment of enumerators, we also aim to experimentally measure and elicit enumerators’ honesty and effort levels. These lab-in-the-field exercises will be administered to all enumerators and supervisors at the beginning of the surveys. All experimental sessions will be conducted in a manner that allows privacy and ensures adherence to the protocol and consistency across participants.

To measure enumerator honesty, we use a private coin-tossing task. The design follows the approach of Barfort et al. (2019) and Fischbacher and F¨ollmi-Heusi (2013), adapted to the survey context. For logistical simplicity, we implement a coin-tossing task instead of a dice-rolling task, as in related studies (e.g., Abeler et al., 2014; Cohn et al., 2014; Janezic and Gallego, 2020). In each round, participants privately toss a coin and report the realized outcome. Reporting heads yields a monetary payoff, while reporting tails yields zero. As the realized outcome is privately observed and payment depends only on the reported outcome, participants have a monetary incentive to misreport realized outcomes by overstating the number of heads.

We elicit effort using an incentivized zero-counting task administered before fieldwork. The task rewards careful attention and perseverance in a routine but demanding exercise (Abeler et al., 2011; Charness et al., 2018; Koch and Nafziger, 2020; Kaiser et al., 2024). The zerocounting exercise is a widely used and incentive-compatible measure of real effort (Charness et al., 2018). This task is repetitive and routine tasks that resemble key features of survey data collection.

Finally, to examine whether enumerators’ personality traits (e.g., Big Five), cognitive (Raven’s test) (Raven et al., 1998), and socio-emotional skills, political opinions, and trust in local and national governments are associated with survey responses, we administer selected modules and questions that are also included in the respondent survey. These include trust in various actors
in communities and governments, as well as related political opinions.

Experimental design

Measuring enumerator honesty
We will conduct a lab-in-the-field experiment with enumerators involved in survey data collection in Ethiopia and Kenya. The design follows the approaches in experimental literature on dishonesty (e.g., Barfort et al., 2019; Fischbacher and F¨ollmi-Heusi, 2013) to measure enumerators’ propensity for dishonesty, and is adapted to our context. For logistical simplicity, we will implement a coin-toss task instead of a dice-rolling task, consistent with prior studies (e.g., Abeler et al., 2014; Cohn et al., 2014; Janezic and Gallego, 2020).

Approximately 80–90 enumerators, including supervisors, will participate across the two countries. Participants will be informed that the task is designed to understand decision-making among survey enumerators engaged in data collection activities and to test how well they guess in situations characterized by randomness. They will also be informed that they earn money based on their reported outcomes. Participants will be explicitly informed that the research team cannot observe the outcome of any coin toss and will be instructed to toss the
coin privately, without the presence of anyone else.

Participants will complete six practice rounds to familiarize themselves with the procedure before starting the actual incentive-compatible exercise. The main coin-tossing exercise and tasks consist of 30 independent rounds. In each round, participants privately toss a coin and
then report their outcome.

The experiment uses local currency. To standardize the task across participants and minimize concerns about coin availability or biased flips from worn or damaged coins, all participants will receive a new coin of the corresponding denomination obtained from a bank. In Ethiopia, participants receive a one-Birr coin, with the lion side defined as heads and the weighing-scale side as tails. In Kenya, participants receive a 20-shilling coin, with the African bush elephant (Ndovu) side defined as heads and the Kenya Coat of Arms (National Crest) side as tails (see Fig. A.1).

Reporting heads pays 50 Birr per round in Ethiopia and 80 KES per round in Kenya, whereas reporting tails pays zero. The Kenya payment is designed to be broadly comparable to the Ethiopian payoff in purchasing-power-parity terms: 50 Birr ≈ 80 KES.2 Since payment
depends only on the reported outcome, participants have a monetary incentive to misreport the realized outcomes by overstating the number of heads.

The true outcomes are not observed by the research team, so this task creates a clear incentive to misreport. Cumulative monetary payments increase with the number of reported heads. Prior studies show that (dis)honesty measured through this approach predicts real-world unethical behavior and corruption (Fischbacher and F¨ollmi-Heusi, 2013; Barfort et al., 2019; Olsen et al., 2019; Cohn and Mar´echal, 2018).

Under truthful reporting, the probability of reporting heads in any given round is 50 percent. Enumerator honesty is measured by comparing the distribution of reported heads to the distribution implied by truthful reporting under random chance. Deviations from this benchmark
distribution will be used to construct measures of enumerators’ propensity to misreport.

Measuring enumerator effort
We measure enumerator effort using an incentivized real-effort task, following Abeler et al. (2011) and Koch and Nafziger (2020). Enumerators count the number of zeros in a sequence of tables consisting of randomly ordered zeros and ones. The sequence of tables is randomly generated for each enumerator, and each table appears one at a time in the CAPI instrument. For each table, enumerators enter the number of zeros and can proceed to the next table only after submitting the correct answer. Incorrect submissions are not counted as completed tables; instead, the enumerator remains on the same table and must try again.

The task lasts for a maximum of 90 minutes, the minimum time needed to complete the field surveys. A countdown timer displayed in the CAPI instrument shows the time remaining. After each correctly completed table, enumerators choose whether to continue counting zeros
in the tables or stop. Enumerators earn 15 Birr for each correctly completed table. To reduce potential confounding, including peer effects, enumerators were randomly assigned to rooms, seated apart from one another, and given staggered start times. These procedures limited
the scope for enumerators to observe others’ progress or stopping decisions. We record the session, room, date, start time, and number of enumerators present, and account for session-level variation in the empirical analysis.

Our primary measure of effort is the number of correctly completed tables before the enumerator stops or reaches the time limit. This measure captures correct work rather than attempted work. The relatively long duration of the task also allows us to examine the pattern of effort over time. We construct additional measures of effort and performance using the task data. As alternative measures of effort provision, we use the total time spent working and accumulated earnings up to the point at which the enumerator stops. As a measure of productivity, we use the average time per correctly completed table.

The task is well-suited for measuring effort in this setting. It requires no prior knowledge, measures performance objectively, and has a limited scope for learning. But the task is tedious and repetitive, and completing additional tables involves costly effort for enumerators, as do conventional surveys.

Design details

Randomization method
The randomization is first done at enumeration or village level to assign enumerators to villages or enumeration areas. The second stage randomization entails random assignment of enumerators assigned for each enumeration area among respondents or a group of respondents. The final unit of randomization varies across surveys.
Randomization unit
Youth groups in Ethiopia and farmers in Kenya
Clusters
42
Observations
42 enumerators administering survey across 183 youth groups in Ethiopia that host about 2500 youth. Similarly, the Kenyan sample includes 40-45 enumerators administering about 2000 households and survey respondents.
Arms
The random assignment involves random assignment of 40-45 enumerators in each survey to about 2000-2500 respondents.
Primary outcomes
1. Enumerator’s real-world preferences and tendencies for unethical behavior and self-reported measures of effort 2. Result of comprehension test 3. Attrition or non-response rates 4. Interview duration or average number of interviews completed 5. Distance covered by enumerators within each village to capture enumerator’s effort.
Secondary outcomes
1. Number of affirmative responses in roaster items or ranking tasks. 2. Objectively measured performance reports by auditors and supervisors.

Cited by

No citation data available for this registration yet.

Keywords