Reinforcing RCTs with Multiple Priors while Learning about External Validity
How to cite
Finan, Frederico; Pouzo, Demian. 2024. "Reinforcing RCTs with Multiple Priors while Learning about External Validity." AEA RCT Registry. AEARCTR-0013634. https://doi.org/10.1257/rct.13634-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This paper presents a framework for how to incorporate prior sources of information into the design of a sequential experiment. These sources can include previous experiments, expert opinions, or the experimenter's own introspection. We formalize this problem using a Bayesian approach that maps each source to a Bayesian model. These models are aggregated according to their associated posterior probabilities. We evaluate a broad class of policy rules according to three criteria: whether the experimenter learns the parameters of the payoff distributions, the probability that the experimenter chooses the wrong treatment when deciding to stop the experiment, and the average rewards. We show that our framework exhibits several nice finite sample theoretical guarantees, including robustness to any source that is not externally valid.
Experimental design
We are working with a large bank in Argentina who is experimenting with different messages to encourage their delinquent clients to refinance their loans. When offering a new refinancing loan, the bank’s first point of contact with its client is via email, and getting the client to click on this email is a critical and necessary first step in the refinancing process. The email informs the client about the basic terms of the contact including the interest rate, which given the current state of the economy might discourage potential customers. An alternative message would be to instead market the contract based on the implied monthly payments. The advantages of the alternative framing are that it avoids potential “sticker shock” and reduces the computational costs for the client.
To test the effects of the different framings, the bank has been conducting a randomized control trial in which delinquent clients are randomized into one of two arms: (1) clients receive information about the contract rate (i.e. status quo messaging), (2) clients receive information about the monthly payments. Using the same treatments, we have asked the bank to conduct a sequential experiment instead of a standard RCT.
Design details
Cited by
No citation data available for this registration yet.