Predicting Biased Polls

How to cite

Brownback, Andy; Gagnon-Bartsch, Tristan; Burke, Nathaniel. 2020. "Predicting Biased Polls." AEA RCT Registry. AEARCTR-0005186. https://doi.org/10.1257/rct.5186-3.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Andy Brownback* , Tristan Gagnon-Bartsch , Nathaniel Burke

* Primary investigator

Registry ID
AEARCTR-0005186
DOI
https://doi.org/10.1257/rct.5186-3.1
First registered
Last updated
Status
Completed
Trial dates
2020-02-11 – 2020-06-01
Countries
United States of America ()
JEL codes
D9, D7
Registry record
http://www.socialscienceregistry.org/trials/5186

Abstract

Unincentivized surveys of stated-preference may be susceptible to "socially desirable responding" (SDR, hereafter) where respondents strategically conceal preferences that they perceive to be socially undesirable. In contrast, economists often assume that the incentive-compatibility of revealed-preference elicitations mitigates the threat of SDR by imposing real costs. One domain in which SDR appears particularly powerful is the realm of politics where SDR is both ubiquitous and well-known by the popular press and other "consumers" of polling research. Moreover, incentive-compatible elicitations of political preferences are not feasible. However, if awareness of SDR is sufficient to de-bias polling information, then SDR may not cause any negative consequences. This project seeks to test the ability of economic agents to de-bias polling data subject to SDR.

In an experimental framework, we will test the accuracy of subjects tasked with predicting a series of choices potentially subject to SDR. Our experiment will have two treatments. In Treatment 1, subjects will observe unincentivized, stated-preference data. In Treatment 2, subjects will observe incentivized, revealed-preference data. While Treatment 1 will be subject to both SDR and sampling errors, Treatment 2 will only be subject to sampling errors. Thus, differences in accuracy will be attributable to differences in the magnitude of bias from SDR. With perfect sophistication, subjects in the two treatments should be equally accurate. However, with partial or fully naive subjects, predictions from Treatment 1 will manifest some of the SDR from the underlying data and will be less accurate. We will measure accuracy in terms of the magnitude and direction of the prediction errors.

Intervention

Subjects in our experiment will be making predictions about a series of choices potentially subject to SDR. Different subjects will receive different information prior to making their predictions.

Experimental design

Subjects in our experiment will be making predictions about a series of choices potentially subject to SDR. Different subjects will receive different information prior to making their predictions.

Design details

Randomization method
To divide subjects into the Hypothetical and Incentivized Choice groups, we will randomly divide our subject pool into two parts and recruit subjects for the groups separately. Recruitment will be identical for the two groups. Subjects will enroll through the SONA recruitment system. At the prediction stage, Qualtrics will randomly assign subjects to receive information based on the Hypothetical or Incentivized Choice groups. We will block "Predictors" based on their prior experience (Hypothetical Choice group, Incentivized Choice group, or neither) This process will be mechanically random and balanced in sample sizes.
Randomization unit
Each subject will be randomly assigned to receive information from the Hypothetical or Incentivized choice groups.
Clusters
800 subjects
Observations
300 subjects (with 1000 extra in a planned extension)-- 100 Hypothetical Choice subjects from the University of Arkansas 100 Incentivized Choice subjects from the University of Arkansas 100 new subjects from the University of Arkansas who make predictions only (planned extension) 1000 subjects from Amazon's Mechanical Turk who make predictions only
Arms
150 subjects receiving information from the Hypothetical Choice group. 150 subjects receiving information from the Incentivized Choice group. (Planned extension) 500 subjects receiving information from the Hypothetical Choice group. 500 subjects receiving information from the Incentivized Choice group.
Primary outcomes
Our primary outcomes of interest are: The accuracy of the final predictions: Both the direction and magnitude of the difference between the prediction (Prior or Posterior) and the truth. The weight placed on each source of information: How much subjects update their predictions based on the information received. We will look at the differences in each of these outcomes based on: Receiving information from the Hypothetical or Incentivized Choice groups Whether the information you received was in the socially-desirable direction The interaction of information source and social-desirability. Being previously assigned to the Hypothetical Choice group, Incentivized Choice group, neither group (but from the University of Arkansas), or neither group (recruited from MTurk) Whether the information you received was from the treatment to which you were assigned
Secondary outcomes
We will also evaluate unincentivized predictions about actions that may be subject to SDR, but we will not observe directly (e.g. unsafe sex). These predictions will be part of a calibration exercise where we use the observed ability of subjects to de-bias information from the Hypothetical choice group to improve predictions about actions subject to SDR.

Cited by

No citation data available for this registration yet.

Keywords