Experts can be biased, a potential reason for the "crisis of expertise" we are experiencing. Yet it is unclear why bias should affect updating. Indeed, if the bias of an expert is well understood, a receiver could ``debias'' their message and recover a useful underlying signal. However, debiasing may be a cognitively complex process for individuals to undertake. We design a lab experiment to understand how the complexity of debiasing information affects its usage and valuation.
Intervention
This experiment will be lab-based, and will take place over Prolific and Qualtrics. All treatments within this experiment will be within participant. Participants will receive information that will aid them in solving a "balls and urns" task. The treatment dimensions for this will be (a) the bias of the piece of information and (b) the complexity of debiasing that information.
Experimental design
Our experiment will be hosted on Qualtrics, and we will recruit participants via Prolific. Participants in our experiment will complete a series of "balls and urns" tasks, in which they guess the probability that balls were drawn from a certain urn. They will then be given valuable information to solve these tasks, after which they can "re-update" their answer. We deliver this information to them with some bias, but tell them exactly how much the information is biased by. We deliver the bias of the information with varying levels of complexity (which we operationalize via algebraic statements of varying length).
Design details
Randomization method
All randomization takes place using Javascript code implemented through Qualtrics.
1) In the main task (described in Part 2 above), the levels of bias that individuals will see are {-10, -5, -1, 0, 1, 5, 10}. There are three levels of complexity levels for each bias level, and 2 expressions for each complexity level above 1. For example, for bias level = 0, we have complexity level 1 = "0%", complexity level 2 = "4 ∗ (2 – 2)/(2 ∗ 2)%" or "6 ∗ (3 – 3)/(2 ∗ 3)%", and complexity level 3 = "2^2 – 4 + 4 ∗ (2 – 2)/(2 ∗ 2) – 8 + 4 ∗ 2%" or "2^3 – 8 + 6 ∗ (3 – 3)/(2 ∗ 3) – 12 + 4 ∗ 3%." All other bias levels are analogous to this.
We will not have a full factorial design when it comes to the crossing of Bayes' posteriors with biases. For the reasoning behind this, consider the following: if Bayes' posterior were, say, 99%, then stating that the computer was underestimating by 5% would imply the true value was 104%, an impossibility. Thus for each value of Bayes' posterior, the code picks from all aforementioned bias values such that the information given is between 0% and 100%.
2) In the willingness to pay section, we will oversample bias = 0, so that it is selected 50% of the time in expectation (and one of the values -10, -5, -1, 1, 5, 10 are selected the other 50% of the time). Furthermore, we will only select from complexity level 1 with 50% probability and complexity levels 2 and 3 with 50% probability.
Randomization unit
Randomization is within participant. So the same participant will in likelihood receive messages of varying (a) bias and (b) complexity.
Clusters
This study does not have clustered randomization.
Observations
We will recruit 500 participants. For our practice updating problems, we will have 3 observations per participant. For our main updating problems, we will have 11 observations per participant. For our willingness to pay problems, we will have 10 observations per participant.
Arms
Randomization takes place via JavaScript code during the study, so we cannot pre-specify exact treatment arm sizes. But we do know the proportions of these treatments arms in expectation.
However, in the main task, we have seven bias values [-10, -5, -1, 0, 1, 5, 10] and three complexity levels [1, 2, and 3]. These 7 x 3 = 21 treatments should be evenly distributed in expectation for questions 2-11 of the main task. For the first question of the main task, we will always use complexity level 1.
In the willingness to pay section, for biases, we will pick 0 with 50% probability, and some value in [-10, -5, -1, 1, 5, 10] with 50% probability. For complexity levels, we will pick complexity level 1 with 50% probability, and from [2, 3] with 50% probability.
Primary outcomes
Our primary outcomes are
a) belief updating (that is, the difference between an individual's prior belief that balls were drawn from an urn, and their posterior belief that balls were drawn from an urn)
b) Willingness to pay for information (that is, a monetary value that the participant said they would be willing to pay for a piece of information)
Secondary outcomes
We will mainly focus on the outcomes above, but plan to look at heterogeneity in these outcomes by
a) baseline skill
b) baseline cognitive uncertainty
c) number of balls drawn in the willingness to pay problems
Cited by
No citation data available for this registration yet.