Endogenous Information Acquisition in Candidate Evaluation
How to cite
Coffman, Katherine; Saygin, Perihan; Kostyshak, Scott. 2023. "Endogenous Information Acquisition in Candidate Evaluation." AEA RCT Registry. AEARCTR-0010969. https://doi.org/10.1257/rct.10969-2.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This project examines how decision-makers acquire and use information in their evaluation decisions. Using a controlled experiment, we explore how individuals assess candidates given noisy signals of quality and how they choose to acquire additional (costly) information about the candidate’s quality before making a final decision. In particular, we ask whether candidate gender impacts evaluator decisions about whether to seek out more information before making a decision. Do evaluators require fewer positive signals to decide that a male candidate is above-the-bar compared to the number of signals they would need to make that same determination for a female candidate? How many negative signals are required before a candidate is rejected, and does this depend on candidate gender? In addition, by comparing across two randomly-assigned treatments, one in which additional information is provided exogenously and one in which decision-makers choose when to acquire more information, we ask whether the endogeneity of information acquisition amplifies discriminatory outcomes.
Experimental design
This is an online study conducted on Prolific. The main task of a participant is to classify candidates as above or below-the-bar.
First, we introduce participants to the task of evaluation of candidates. These candidates are past study participants who completed a math and science test and were classified as above or below-the-bar based upon having a test score above a set threshold. We use these past study participants to construct “pools” of candidates for our study participants to assess.
Before assessing candidates, each evaluator (main study participant) is provided with information about the candidate pool from which their candidates are chosen. In particular, they are told the share of male candidates who were above-the-bar (likelihood of a given male candidate being above-the-bar) and the share of female candidates who were above-the-bar.
Evaluators are randomly-assigned to one of four different candidate pools. There are 50 total candidates in each pool. The pools vary in the likelihood of male and female candidates being above-the-bar:
(1) 40-40 pool: 40% of female candidates and 40% of male candidates were above-the-bar
(2) 60-60 pool: 60% of female candidates and 60% of male candidates were above-the-bar
(3) 40-60 pool: 40% of male candidates and 60% of female candidates were above-the-bar
(4) 60-40 pool: 60% of male candidates and 40% of female candidates were above-the-bar
After describing the selected pool to the evaluator, we randomly-draw a candidate from the described pool for the evaluator to assess. Participants see some basic information about this candidate, including their gender and their answers to some (arguably irrelevant) questions (i.e., do you prefer mountain vacations or beach vacations?). Participants provide an incentivized estimated likelihood of that candidate being above-the-bar.
Participants can pay to acquire additional information about the candidate. We refer to these as “signals” – a signal can be either good or bad. The probability with which a good signal is drawn depends upon the true performance of the candidate:
• A candidate who is above-the-bar produces a good signal with probability of 75% and a bad signal with probability of 25%
• A candidate who is below-the-bar produces a good signal with probability of 25% and a bad signal with probability of 75%.
Our primary treatment randomization is whether evaluators can choose whether or not to acquire signals about the candidate. We call this endogenous or exogenous information acquisition:
• Exogenous information acquisition: the evaluator must acquire 5 signals about the candidate before making a classification. After each signal, they provide an updated incentivized likelihood of the candidate being above-the-bar. After acquiring all 5 signals, they make a final binary classification of the candidate, classifying the candidate as either “above-the-bar” or “below-the-bar.”
• Endogenous information acquisition: the evaluator can choose to acquire up to 5 signals about the candidate before making a classification. After each signal, they provide an updated incentivized likelihood of the candidate being above-the-bar. At any point, the evaluator can choose to stop acquiring signals and make a final classification, classifying the candidate as either “above-the-bar” or “below-the-bar.”
Both the random assignment to candidate pools and the random assignment to information treatment is across-participant. All evaluators evaluate 5 candidates from their selected pool, chosen randomly with replacement. They receive no feedback or information on their decision-making until the end of the study.
We will pay 1 out of every 10 participants (randomly-selected) additional incentive pay. For those participants:
At the conclusion of the study, we randomly select one candidate that participant evaluated. For that randomly selected candidate, we pay a bonus of $7 if their final classification of the chosen candidate is correct, and a bonus of $0.50 if they incorrectly classified the randomly-chosen candidate. In addition, we deduct $0.05 from the bonus payment for every signal they acquired for the randomly-chosen candidate in the endogenous treatment. We also randomly select one likelihood slider and pay them an extra $1 if within 10pp of the true likelihood.
Design details
Cited by
No citation data available for this registration yet.