Replication: Belief elicitation and behavioral incentive compatibility

How to cite

Lehmann, Niklas Valentin. 2025. "Replication: Belief elicitation and behavioral incentive compatibility." AEA RCT Registry. AEARCTR-0016853. https://doi.org/10.1257/rct.16853-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Niklas Valentin Lehmann*

* Primary investigator

Registry ID
AEARCTR-0016853
DOI
https://doi.org/10.1257/rct.16853-2.0
First registered
Last updated
Status
Completed
Trial dates
2025-10-03 – 2025-10-19
Countries
Germany ()
Registry record
http://www.socialscienceregistry.org/trials/16853

Abstract

Danz, Vesterlund, and Wilson (2022) demonstrate in their article "Belief elicitation and behavioral incentive compatibility" that the binarized scoring rule, a widely-used state-of-the-art mechanism for belief elicitation, leads to systematically distorted reports. We replicate their experiment to validate this effect and gather supplementary data to probe additional explanations for the bias. Furthermore, our study assesses the robustness and generalizability of their findings to broader settings.

Experimental design

This study investigates how participants report their probabilistic beliefs. We examine the accuracy and honesty of belief reporting, as well as participants' understanding of the tasks.

Design details

Randomization method
Randomization is achieved on the server side. Participants are assigned to treatment or control based on the assigned participant ID (odd or even).
Randomization unit
We randomize on the participant level.
Clusters
We attempt to recruit 2×70 participants.
Observations
The planned number of observations is 140. However, we remark that we recruit participants during a conference (closed pool). Thus, we do not continue to sample new participants until the planned number is reached, and the actual number of observations may be very different.
Arms
We attempt to recruit 70 participants each in two treatment arms.
Primary outcomes
We will measure the outcomes reported in the original study by Danz et al. (2022), which primarily include the rate of false reports and the proportions of reports that pull to center, to the distant extreme, and to the near extreme. These metrics will be calculated based on participants’ main five predictions. Additionally, we will record participants’ choices in the “incentives only” tasks, specifically whether each choice is compatible with the dominant strategy and, among incompatible choices, whether responses pull to center, near extreme, or distant extreme. We will report the fraction of incompatible choices in each of these categories. Furthermore, diverging from the original study, we will also assess the accuracy, calibration, and precision of participants’ predictions. Primary outcomes will be compared across treatment groups.
Secondary outcomes
We will measure participants’ self-assessed understanding of the tasks and perceived honesty in reporting. We will examine whether confusion or lack of understanding affects primary outcomes, and assess whether the experimental treatments influence understanding and honesty. Additionally, we will record the time participants spend on each task as a proxy for effort.

Cited by

No citation data available for this registration yet.

Keywords