Danz, Vesterlund, and Wilson (2022) demonstrate in their article "Belief elicitation and behavioral incentive compatibility" that the binarized scoring rule, a widely-used state-of-the-art mechanism for belief elicitation, leads to systematically distorted reports. We replicate their experiment to validate this effect and gather supplementary data to probe additional explanations for the bias. Furthermore, our study assesses the robustness and generalizability of their findings to broader settings.
Experimental design
This study investigates how participants report their probabilistic beliefs. We examine the accuracy and honesty of belief reporting, as well as participants' understanding of the tasks.
Design details
Randomization method
Randomization is achieved on the server side. Participants are assigned to treatment or control based on the assigned participant ID (odd or even).
Randomization unit
We randomize on the participant level.
Clusters
We attempt to recruit 2×70 participants.
Observations
The planned number of observations is 140. However, we remark that we recruit participants during a conference (closed pool). Thus, we do not continue to sample new participants until the planned number is reached, and the actual number of observations may be very different.
Arms
We attempt to recruit 70 participants each in two treatment arms.
Primary outcomes
We will measure the outcomes reported in the original study by Danz et al. (2022), which primarily include the rate of false reports and the proportions of reports that pull to center, to the distant extreme, and to the near extreme. These metrics will be calculated based on participants’ main five predictions.
Additionally, we will record participants’ choices in the “incentives only” tasks, specifically whether each choice is compatible with the dominant strategy and, among incompatible choices, whether responses pull to center, near extreme, or distant extreme. We will report the fraction of incompatible choices in each of these categories.
Furthermore, diverging from the original study, we will also assess the accuracy, calibration, and precision of participants’ predictions.
Primary outcomes will be compared across treatment groups.
Secondary outcomes
We will measure participants’ self-assessed understanding of the tasks and perceived honesty in reporting. We will examine whether confusion or lack of understanding affects primary outcomes, and assess whether the experimental treatments influence understanding and honesty.
Additionally, we will record the time participants spend on each task as a proxy for effort.
Cited by
No citation data available for this registration yet.