Adaptive Treatment Assignment for Policy Choice: Phone Enrollment for an Agricultural Extension Service

How to cite

Sautmann, Anja; Kasy, Maximilian. 2019. "Adaptive Treatment Assignment for Policy Choice: Phone Enrollment for an Agricultural Extension Service." AEA RCT Registry. AEARCTR-0004263. https://doi.org/10.1257/rct.4263-2.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Anja Sautmann* , Maximilian Kasy

* Primary investigator

Registry ID
AEARCTR-0004263
DOI
https://doi.org/10.1257/rct.4263-2.1
First registered
Last updated
Status
Completed
Trial dates
2019-06-03 – 2019-08-31
Countries
India ()
JEL codes
C93, C11, O22, O13
Registry record
http://www.socialscienceregistry.org/trials/4263

Abstract

The purpose of this experiment is to test the implementation of an adaptive experimental procedure designed by the authors in a real-world context.
The goal of many experiments is to inform the choice between different policies. However, standard experimental designs are geared toward point estimation and hypothesis testing. We consider the problem of treatment assignment in an experiment with several cross-sectional waves where the goal is to choose among a set of possible policies (treatments) for large-scale implementation. We show that optimal experimental designs learn from earlier waves by assigning more experimental units to the better-performing treatments in later waves. We propose a computationally tractable approximation of the optimal design, based on a modification of Thompson sampling.
We collaborate with a non-profit organization that supports smallholder farmers in developing countries by providing personalized agricultural advice through mobile phones in India. We apply the proposed adaptive trial design to help the non-profit choose efficiently among several alternative call procedures (treatments) to enroll farmers into their service. Since the adaptive design is based on Bayesian optimization, it allocates more subjects to the most successful treatment arms in consecutive study waves, which improves learning and at the same time applies the more successful treatment variants earlier to more subjects.

Intervention

Precision Agriculture for Development (PAD) is testing interactive voice response (IVR) phone calls with the purpose to enroll as many rice farmers as possible into a farmer information service. The main content of the call consists in a set of enrollment questions and does not vary across treatments.
The experiment consists in varying how these phone calls are initiated. Possible treatment variants considered involved text messages prior to the call to announce the call time, starting the call with a jingle, calling at different times of the day, etc.
The final choice of treatments selected by PAD for testing was:
- sending no text message to announce the call, sending a text message 1 hour before, and sending a text message 24 hours before;
- calling farmers at 10 am, and calling farmers at 7pm.
Text messaging and call times were cross-randomized for a total of 6 treatment arms.
Each study wave takes two days (one day to send text messages, one day to carry out calls).

Experimental design

The study is conducted in India, with farmers who are potential clients of Precision Agriculture for Development, an NGO that attempts to improve the lives of farmers by providing tailored farming advice.
Precision Agriculture for Development (PAD) is testing a variety of different interactive voice response (IVR) treatments. The purpose of these treatments is to enroll as many rice farmers as possible into the farmer information service. PAD designed these treatments based on pilots and experience from other enrollment campaigns. The treatments are fully designed and implemented by PAD and their final choice was to test six distinct treatment arms (see above). The adaptive experimental procedure the authors designed helps PAD to choose the treatment that works best for their setting.
The experiment is conduced in waves of 600. The only information PAD has are a very large set of phone numbers provided by an outside party. An employee of PAD processes these numbers by querying if they are valid, and if they are on a do-not-disturb list. Only valid numbers not on the list are then used. PAD is setting aside 10'000 phone numbers that are up for calling in the target period for this experiment.
Experimental waves are carried out consecutively, where each wave takes two days to administer (text messages sent up to 24 hours ahead of time, two call times in morning and evening on the call day). In the first wave, 100 phone numbers are randomly selected from the set of 10'000 numbers and each is assigned randomly to one of the treatment arms. The outcomes of each experimental wave are submitted to an app programmed by the authors. The app uses treatment success rates in each treatment arm to update a flat prior on possible average success rates, and applies the Exploration Sampling algorithm to choose the number of phone numbers to be assigned to each treatment arm in the next wave. A new wave of 600 phone numbers are then randomly selected and assigned to treatment arms. Exploration Sampling is adapted from standard Thompson sampling for waves of size 1 by (a) reducing sampling variation by taking advantage of batch sampling (of 600 units per wave here), and (b) re-distributing experimental units from the best performing option to close competitors to improve learning. The experiment continues until 10'000 phone numbers are used.

Design details

Randomization method
Computer-based randomization.
Randomization unit
Individual phone number.
Clusters
600 (or less) per wave up to a total of 10'000 numbers.
Observations
600 (or less) per wave for a total of 10'000 numbers.
Arms
100 per wave in expectation. Number of units in each arm is chosen as part of the experimental design during the trial. Final sample size approved by IRB may be up to 10'000.
Primary outcomes
Successful call completion: binary variable describing whether the call recipient answered five IVR questions asked during the call (answers to these questions enable processing).
Partners
Name: Precision Agriculture for Development Partner_type: ngo Url: https://precisionag.org/

Materials

Papers from this trial

  1. Kasy, Maximilian and Anja Sautmann, 2021. Adaptive treatment assignment in experiments for policy choice. Econometrica 89(1), 113-132. https://doi.org/10.3982/ECTA17527

Cited by

No citation data available for this registration yet.

Keywords