Prolific sample for: Competing Causal Interpretations: An Experimental Study

How to cite

Ambuehl, Sandro; Thysen, Heidi. 2023. "Prolific sample for: Competing Causal Interpretations: An Experimental Study." AEA RCT Registry. AEARCTR-0012652. https://doi.org/10.1257/rct.12652-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Sandro Ambuehl* , Heidi Thysen

* Primary investigator

Registry ID
AEARCTR-0012652
DOI
https://doi.org/10.1257/rct.12652-1.1
First registered
Last updated
Status
In development
Trial dates
2023-12-05 – 2023-12-16
Countries
Private
JEL codes
C91, D01, D83
Registry record
http://www.socialscienceregistry.org/trials/12652

Abstract

This is a follow up to a lab study we conducted previously. The abstract of the lab study is this:

A central factor when choosing an action is its causal effect on the outcome of interest. Yet, causal information is often lacking. People instead observe correlational or historical data, along with causal interpretations and recommendations provided by experts who frequently disagree with each other. Our laboratory experiments study choice in such settings, where beliefs concern the structure of the data-generating process rather than merely magnitudes. Roughly half of our subjects attempt to determine the fit of the causal interpretations to past data, as the literature on model persuasion assumes. We characterize the limits to their ability to do so. Half the subjects’ choices are co-determined by the interpretations’ promises of future payouts, as the literature on narrative competition assumes, or by the downside these choices entail if they are mistaken. Subjects also commonly employ heuristics such as Occam’s razor. The fact that they typically prefer flexibility over parsimony insures them against bad choices in some settings but has the opposite effect in others. Our estimates predict well out of sample and closely agree across two different identification strategies in two different samples. We also study the extent to which behavior is robust to framing and the relation between subjects’ choices and their political attitudes and psychological characteristics. Our results characterize the cases in which subjects’ behavioral tendencies render them most receptive to misleading interpretations. They also inform the literatures on narrative competition and model persuasion.

Intervention

SETTING: See description on experimental design

INTERVENTION: We run two treatments. In one treatment, we provide help to subjects, in two forms. First, subjects have access to a description of the correlational implications of causal structures. Second, we highlight data charts for which the competing interpretations have conflicting implications. In the other treatment, subjects get no such help.

Experimental design

In our experiment, subjects choose an action at a cost. The action possibly affects a monetary outcome. Whether it does and by how much depends on a data-generating process (DGP) that involves the action, the outcome, and an additional quantity (the covariate). Subjects do not know the DGP, but they observe information about it. Specifically, in each round, we fit a misspecified model to large datasets generated by the DGP, along with the true model. For each model, we calculate the implied optimal action (the recommendation), and the payoff the subject can expect if the specification is correct and the subject follows that model (the promise). Subjects observe each model’s recommendation and promise. While we do not tell subjects which models are misspecified, we present them with a graphical description of each model in the form of a directed acyclic graph (DAG). In addition, subjects can observe unconditional correlations between any pair of variables in the data generated by the DGP, as well as the correlations between any pair of variables conditional on any third variable. Subjects who wish and are able to do so can therefore check which models may be misspecified. Throughout, our DGPs and misspecified models are recursive systems of linear Gaussian equations. Subjects choose which model to follow. The corresponding action recommendation is carried out and determines the subjects’ payout according to the DGP.
We frame this setting to subjects as follows. A true ‘mechanism’ determines the relation between the subject’s ‘action’ and her ‘bonus.’ Two or three ‘advisors’ each have a ‘theory’ about that mechanism. While data cannot affect an advisor’s theory, each advisor has used the data generated by the true mechanism to derive a recommendation for what action is optimal and what bonus the subject can expect if the advisor’s theory is correct. Subjects select one of the advisors whose recommendation is carried out.

Design details

Randomization method
Randomization done online by computer
Randomization unit
Subject
Clusters
Total of 800 subjects, 400 for each treatment.
Observations
Each subject makes 18 main choices between conflicting interpretations
Arms
Total of 800 subjects, 400 for each treatment.
Primary outcomes
The sequence of chocie of interpretations across all rounds of the study.
Secondary outcomes
Relation of choice of interpretations to demographic factors, especially education

Cited by

No citation data available for this registration yet.

Keywords