On the Fairness of Machine-Assisted Human Decisions
How to cite
McLaughlin, Bryce; Gillis, Talia; Spiess, Jann. 2022. "On the Fairness of Machine-Assisted Human Decisions." AEA RCT Registry. AEARCTR-0010416. https://doi.org/10.1257/rct.10416-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We are studying how the structure of information presented to human decision-makers affects their performance and bias in a prediction task.
Specifically, study subjects predict the performance on math tasks of multiple test-takers from a previous experiment. For each test-taker, the study subjects see (i) test-taker characteristics (age, gender, education) as well as (ii) assistance in the form of an average of the performance of other test-takers with similar characteristics.
The main intervention is variation in the information by which the performance of previous test-takers is averaged. In the treatment conditions, averages are formed separately by gender. In the control conditions, averages are taken jointly across genders.
The main outcome of interest is the average difference in assessments of women relative to men. We hypothesize that supplying averages that vary by gender increase the predictions of women’s performance relative to predictions of men’s performance.
Study subjects are US adults recruited through Prolific. We will report summary statistics of age, gender, and education characteristics with our data analysis.
Conditions are assigned randomly at the subject level. There are three treatment conditions and three corresponding control conditions. The total target sample size is 1250.
Intervention
We change the information available to decision-maker's in a skills evaluation task by adjusting an assistant which feeds them conditional averages.
Experimental design
Study subjects will look at a series of up to 12 randomly ordered profiles of test-takers from a previous experiment who took a six-question math test and estimate their performance on the test. (Profile data is from Cecelia Ridgeway of Stanford University and Tamar Kricheli-Katz of Tel Aviv University for their upcoming work “Behavioral responses to the changing world of gender.” We received the data directly from the authors; it is de-identified and is not publicly available.) Participants are rewarded based on the accuracy of their predictions proportional to the square of the deviation, so that predicting their best guess (posterior expectation) is theoretically optimal. For each test-taker, study subjects are shown the test-taker’s gender g, and two additional covariates, x1 (age) and x2 (having acquired a four year college degree).
All study subjects also receive baseline information about the composition of test-takers and overall average performance. Subjects are randomized into one of six conditions, which vary by which additional summary information (based on a training sample of test-takers that are not shown to subjects in the experiment) they receive for each of the test-takers they review:
Condition 0−: Participants receive average score of all training test-takers.
Condition 0+: Participants receive the average score of test-takers in the training sample who share g with the profile they are evaluating.
Condition 1−: Participants receive the average score of test-takers in the training sample who share the same age (x1) range with the profile they are evaluating.
Condition 1+: Participants receive the average score of test-takers in the training sample who share the same age (x1) range and g with the profile they are evaluating.
Condition 2−: Participants receive the average score of test-takers in the training sample who share the same age (x1) range and x2 with the profile they are evaluating.
Condition 2+: Participants receive the average score of test-takers in the training sample who share the same age (x1) range and x2 and g with the profile they are evaluating.
After submitting their evaluation, participants will answer questions about their beliefs in their own prediction abilities as well as to what extent they adjusted their predictions in response to prediction provided (their trust in that signal). Participants will also fill in their own gender g and covariates x1 and x2 to allow us to check for in-group biases. Each condition is also divided into two sets (same across conditions). One set is a gender flip of the other so we obtain balanced groups when comparing the evaluation of women and men. Target sample sizes (subjects) are 375 in 1+ and 1− each, and 125 in each of the others.
Design details
Cited by
No citation data available for this registration yet.