Do Policymakers Distinguish Correlation from Causation? Evidence from an international Survey Experiment

How to cite

Keeble, Jack. 2026. "Do Policymakers Distinguish Correlation from Causation? Evidence from an international Survey Experiment." AEA RCT Registry. AEARCTR-0019081. https://doi.org/10.1257/rct.19081-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jack Keeble*

* Primary investigator

Registry ID
AEARCTR-0019081
DOI
https://doi.org/10.1257/rct.19081-1.0
First registered
Last updated
Status
On going
Trial dates
2026-06-09 – 2027-03-31
Countries
Private
JEL codes
C91, D91, H11
Registry record
http://www.socialscienceregistry.org/trials/19081

Abstract

This study examines whether policymakers distinguish between correlational and causally identified evidence when evaluating policy-relevant research. Policymakers recruited through the Warwick Policymaker Lab complete an online experimental vignette survey in which they evaluate four summaries of evidence on the same healthcare intervention. The vignettes vary experimentally along two dimensions: the inferential basis of the evidence (correlational versus random assignment) and the geographical context (the respondent's own country versus a similar foreign country). The primary outcome is respondents' assessment of the causal credibility of the evidence, while the secondary outcome is their stated willingness to rely on the evidence for policymaking. The primary analysis exploits the randomisation of the first vignette to estimate between-subject treatment effects. Secondary analyses use all four vignette evaluations in respondent fixed-effects models and examine heterogeneity by methodological training, policymaking experience, policy sector, and country where sample sizes permit.

Experimental design

Online survey experiment with policymakers. Respondents evaluate short evidence vignettes about the same policy intervention. Vignettes vary along two dimensions: inferential basis of the evidence, correlational versus random assignment, and geographical context, own country versus similar foreign country. Vignette order is randomised at the respondent level. The analysis uses the first vignette as a between-subject comparison and all four vignettes in within-subject specifications.

Design details

Randomization method
Randomisation done by computer within the oTree survey platform. The order of the four evidence summaries is randomly assigned separately for each respondent.
Randomization unit
Individual respondent. Each respondent is randomly assigned an order in which to view the four evidence summaries.
Clusters
Up to 250 individual respondents.
Observations
Up to 250 individual respondents, generating 4 respondent-vignette observations each.
Arms
For the first-vignette between-subject analysis, respondents are expected to be approximately evenly distributed across the four vignette conditions: correlational/own country, random assignment/own country, correlational/similar foreign country, and random assignment/similar foreign country. For the within-subject analysis, each respondent evaluates all four conditions.
Primary outcomes
Causal convincingness: respondent rating of how convincing the evidence is that text-message reminders led to faster discharge times, measured on a 7-point scale from 1 = not at all convincing to 7 = very convincing.

Cited by

No citation data available for this registration yet.

Keywords