Understanding and Improving Policymakers' Sensitivity to Program Impact

How to cite

Toma, Mattie; Bell, Elizabeth. 2021. "Understanding and Improving Policymakers' Sensitivity to Program Impact." AEA RCT Registry. AEARCTR-0007659. https://doi.org/10.1257/rct.7659-3.3000000000000003

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Mattie Toma* , Elizabeth Bell

* Primary investigator

Registry ID
AEARCTR-0007659
DOI
https://doi.org/10.1257/rct.7659-3.3000000000000003
First registered
Last updated
Status
Completed
Trial dates
2021-05-24 – 2021-11-05
Countries
United States of America ()
JEL codes
D9, I38, D63, C9
Registry record
http://www.socialscienceregistry.org/trials/7659

Abstract

When making decisions about which programs to implement, policymakers must assess impact in the face of bounded rationality constraints when processing relevant information. These constraints may result in “under-sensitivity” to impact-relevant information about evidence-based programs, ultimately leading to support for less impactful interventions, as compared to a utilitarian benchmark. This study uses a lab-in-the-field experiment among federal employees of the US government to estimate and seek to improve sensitivity. We will first document any under-sensitivities in government decision making across three key program attributes relevant to impact: scope, outcome type, and effect persistence. The primary goal of this study is to test modes of presenting program information that increase sensitivity across these domains, in turn identifying techniques for researchers and evaluators to use to more effectively disseminate results of program evaluations.

Intervention

This project will leverage a lab-in-the-field survey experiment among federal employees in the US government. All survey respondents will be presented with descriptions of hypothetical programs and evaluation results and will estimate the maximum cost at which they would be willing to fund each program. We will vary the mode of presenting the information across programs. Specifically, respondents may see a program description presented with no additional framing ("baseline"); a description with a calculator that translates total program costs into the cost per person affected per year ("impact calculator"); or two similar descriptions presented together on one page ("side-by-side").

Experimental design

Different modes of presenting program information will be randomly assigned within respondents. That is, respondents will see two program descriptions presented using each of the "baseline" condition, the "impact calculator," and the "side-by-side" comparison, in random order. The programs shown for each condition as well as the calculated impact shown for each program will be randomly varied across respondents.

Design details

Randomization method
The randomization will be implemented in Qualtrics.
Randomization unit
Conditions will be randomized within respondent.
Clusters
The originally-specified sample was 500 participants. Recruitment has been more challenging than expected, so we are committing to a date rather than sample cutoff -- the final sample will be the N collected as of October 25, 2021. At the time of this update, N = 121.
Observations
N x 6
Arms
N
Primary outcomes
The primary outcome of interest is respondents' perceived program value, which is defined as the maximum cost at which the respondent would be willing to fund the program, as identified in the experiment.
Secondary outcomes
The secondary outcome of interest is self-reported confidence in one's valuation assessments, as a proxy for cognitive uncertainty.

Papers from this trial

  1. Toma, M. and E. Bell. 2021. "Understanding and Improving Policymakers' Sensitivity to Program Impact," Working Paper. https://drive.google.com/file/d/1qkStG3Y-FZvizfUzQwUP2GkD7hzMlM-I/view?usp=sharing

Cited by

No citation data available for this registration yet.

Keywords