"Addressing the Mismeasure of Women: an Experimental analysis of Biased Proxies and Biased Beliefs"
How to cite
Ward-Griffin, Peter.
2025. ""Addressing the Mismeasure of Women: an Experimental analysis of Biased Proxies and Biased Beliefs"." AEA RCT Registry. AEARCTR-0014564.
https://doi.org/10.1257/rct.14564-1.0
Download:
BibTeX
·
RIS— for reference managers without a browser
translator. Zotero and most
others pick up this page's citation metadata directly.
How can we de-bias estimates of highly skilled women’s ability in hiring and promotion decisions? What interventions are effective at improving people’s ability to evaluate the relative ability of women and men for positions of power? To investigate these questions, I use an online experiment on Indian MTurk users to identify how they make incentivized hiring decisions when presented with limited information, including a biased but informative proxy variable for ability with distributions that differ by gender. I look at how people learn across repeated decisions, and how participants’ previous exposure to women in positions of power or social interactions influences participants’ ability to learn about cross-group differences. For my central interventions, I analyse the extent to which information and active debiasing methods work to address these issues. I further identify whether the anchoring-and-adjustment heuristic impedes learning and whether active debiasing serves to overcome it. In addition, I compare how learning differs when people are aware of the job candidate's gender versus gender-blind hiring practices. This experiment aims to improve our understanding of the extent to which people fail to understand differential signal mappings across sub-groups, the impact of treatments to enhance that understanding, whether people can learn through repeated exposure, or whether differential signal neglect is a persistent phenomenon even with incentives to learn.
I theorize that people exhibit a "differential signal neglect" more broadly — they fail to recognize that the same observable score predicts different outcomes for different groups. Even when given information about bias patterns or adjustment tools, decision-makers anchor on mean differences and insufficiently adjust for the full complexity of group-specific signal-to-outcome mappings. This learning barrier parallels systematic biases in econometric practice where researchers control for group means while ignoring higher-order distributional differences. Relatedly, widespread research methods in various strands of literature in economic, political, and statistical sciences incorrectly incorporate actual differences (such as in cases of differential selection) into the adjustment and fail to provide unbiased estimates. I argue that existing methods to identify the impact of policies on ability apply incorrect inference procedures, which lead to inconsistent estimates of women's ability relative to men. Incomplete or inaccurate adjustment procedures are especially concerning as we most underestimate the impact of gender-based affirmative action policies when they are most effective. Failure to adjust for distributional disparity distorts policy analyses even more because of the similar failure to account for selection in many applied economic inference approaches. Together, these misestimation tendencies lead us to systematically underestimate the benefits of quotas and affirmative action policies by incorrectly incorporating the average difference in ability due to selection into the gender dummy. It seems plausible that decision-makers in applied economic settings would display a similar bias. Failure to learn about the disparity in the relationship between actual productivity and interview scores is especially likely, as these decision-makers (relative to researchers) have less exposure to distributions and have less salient information shocks. This bias may influence regulatory and fiscal policy decisions, including gender quota implementations and evaluations, affirmative action plans, and assessment of educational initiatives, as well as firm hiring, university-level admittance decisions, and court conviction decisions.
Intervention
MTurk Workers complete an incentivized simulated hiring task with limited information, including a relevant but biased proxy variable for the ability of candidates. This scenario is common in real life hiring decisions and in voting behaviour. They are randomly treated with different levels of additional information about the bias over time and receive feedback based on their previous decisions. They also complete a detailed survey task, including opinions about fairness in hiring and previous job experience. The “main treatments” are included in two treatment categories: information provision and active debiasing. Two levels of these treatments and five other treatments are used for disentangling important effects. These will be described in detail upon trial completion. This study aims to evaluate how people learn and make choice decisions in limited information environments with informative but biased signals of latent ability and with different mappings across sub-groups. I will publish more details about the interventions on trial completion. These are pre-registered in the hidden section below.
Experimental design
Participants (Indian MTurk workers aged 25-65) complete a 40-question binary choice task designed to simulate comparison-based hiring tasks using interview scores. For this experiment, I am focusing on numerical interview scores. They are a very widely used score in employment, education, and training settings, and they have two critical features: 1) They are an informative signal of ability 2) They often have significant differences in the sub-group interview score distributions, sometimes even when performance is equal The experiment evaluates the effect of information, salience, and other treatments on participants’ decision-making efficiency and equity. All randomization is conducted using JavaScript and Qualtrics’ internal randomization measures. Participants receive individual choice-level feedback about their decision quality every five decisions, and learning is evaluated. Participants experience a series of cross-randomized information shocks that may influence how they make decisions. This experiment is designed to randomize the salience of information and sub-group category knowledge, as well as population- or sample-level information shocks. The impact of the interaction of the shocks with each other and vital participant characteristics is expected to shape how people make decisions, the proportion of correct choices they make, and may affect critical equity-related outcomes. Separately, treatment conditions' impact and order effects are tested to ensure unbiased estimation of learning and participant outcomes. Half of the participants will also complete a 10-question productivity guessing task. Participants assigned an initial information treatment or gender knowledge are also assigned the same treatments for the guessing task. Participants: N = 1,600 MTurk workers from India Age range: 25-65 years Main Task Structure: Receive information about the past 10 hires from each group If in (subgroup knowledge) condition: know they are seeing 10 past hires of (subgroup of interest) versus (second subgroup of interest) Otherwise, they simply know they are comparing across candidates from college A at a university or college B. Complete 40 Sequential Choices across 8 rounds: Each round: 5 simultaneous comparisons with binary decisions Choose the higher-productivity candidate from each pair After each 5-decision round: See chosen candidates' true productivities Participants receive no feedback on unchosen candidates. More details will be released at the study’s completion.
Design details
Randomization method
The primary form of randomization is using JavaScript that runs on the Qualtrics platform for participants from MTurk. Variants of the Math.random() code are used.
Randomization unit
Randomization is done in a hierarchical structure across and within participants. First, participant groups are assigned a changing group-level for every 10 decisions. Second, participant-level variables are randomized. Third, individual-task elements are randomized.
More precisely, participants are first assigned to individual person-level treatment groups. These include gender knowledge, which is consistent across all comparison tasks and the productivity task, whether they are to complete the guessing task, and the characteristics of the distribution they will be drawing from. This includes population-level gender bias (separated into overestimation of men, underestimation of women, and differential selection), productivity variance, covariance of candidate performance within job types, informativeness of interview scores, candidate average productivity, whether men are on the left or right (tied to whether they are under interviewer A), whether participants receive repeated information, or simply receive information once conditional on receiving information, whether they guess the productivity of candidates interviewed by interviewer A or B in the guessing task, whether they have a social desirability shock, whether they complete the matrices prior to the task or post, and whether they complete the gender-salience survey pre- or post.
Other treatments are randomized at the 10-question cluster level including the main treatments: information about mean discrepancies between interview scores and productivity, (incomplete) information about the slope between interview scores and productivity, and whether participants are provided with adjusted scores that incorporate the information provided to them.
Knowledge is provided at the five-question level, with each participant receiving information about each of their last five hires at the end of each set of five decisions.
At the individual decision level, each participant is influenced by the randomly generated individual interview score for men and women, and aims to evaluate the productivity of each to decide which selection would generate more money for them.
Clusters
There are 1600 people, with 40 choices each, giving a total of 64,000 decisions in 6400 10-decision treatment clusters (and 12,800 five-decision clusters), but analyses will focus on using a mixed effects model with a three-level hierarchy (person>10-decision>individual decision).
Observations
64,000 decisions
Arms
For this study, I recruit N = 1,600 participants from Amazon Mechanical Turk workers located in India. Each participant completes 40 binary hiring decisions between paired candidates (one man and one woman), yielding 64,000 total observations with clustering at the individual level. Based on the pilot data, the number of effective observations for decision-level outcomes will range from 16,196 to 54,000, depending on the ICC of a particular outcome. This reflects a design effect of up to 3.95.
Primary Treatment Assignment
Participants are randomly assigned to one of the 12 experimental conditions through weighted randomization. Each category has allocation probabilities ranging from 1/40 to 6/40. These affect the information and debiasing that participants receive for their initial 10 comparison decisions. Information treatments affect the information that participants receive for their first 10 guessing tasks as well. Participants can move between categories to higher information or adjustment levels.
Initial Treatment Distribution:
Conditions Without Gender Information (Categories 1-6):
Condition 1 (No bias information, No population debiasing): 240 participants
Of these, 60 receive sample debiasing, repeating (and summarizing) information based on the past 10 hire information
Condition 2 (Mean bias information, No debiasing): 200 participants
Condition 3 (Linear prediction information, No debiasing): 120 participants
Condition 4 (Mean bias information, Mean-based debiasing): 120 participants
Condition 5 (Linear prediction information, Mean-based debiasing): 80 participants
Condition 6 (Linear prediction information, Linear debiasing): 40 participants
Conditions With Gender Information (Categories 7-12):
Condition 7 (No bias information, No debiasing): 240 participants
Of these, 60 receive sample debiasing, repeating information based on the past 10 hire information
Condition 8 (Mean bias information, No debiasing): 200 participants
Condition 9 (Linear prediction information, No debiasing): 120 participants
Condition 10 (Mean bias information, Mean-based debiasing): 120 participants
Condition 11 (Linear prediction information, Mean-based debiasing): 80 participants
Condition 12 (Linear prediction information, Linear debiasing): 40 participants
Dynamic Information Provision
The design incorporates progressive information revelation, where participants may receive additional information regarding discrepancies between interview scores and productivity across interviewers or debiasing tools as they progress through the task. Participants can only advance to conditions with more information or stronger debiasing methods, never regress. This transition is given by a JavaScript variable sequence of categories (OriginCat - Original Category, SecondCat- Second Category…) with initial probability weight given by a weighted JavaScript random number generator and then subsequent uniform re-distribution to (weakly) higher categories based on a random variable exceeding the specified threshold. Transition probabilities are:
After decisions 1-10: 65% remain in current condition, 35% may advance
After decisions 11-20: 65% remain in current condition, 35% may advance
After decisions 21-30: 40% remain in current condition, 60% may advance
Expected Distribution Over Time:
Condition
Initial (N)
After 10 decisions
After 20 decisions
After 30 decisions
No information/No debiasing
480
340
240
120
Mean information/No debiasing
400
316
248
152
Linear information/No debiasing
240
240
226
190
Mean information/Mean debiasing
240
240
226
190
Linear information/Mean debiasing
160
244
286
346
Linear information/Linear debiasing
80
220
270
500
Additional Experimental Manipulations
1. Gender Salience Priming
Half of the participants (800) will complete the gender-attitude survey questions before the main hiring task to increase the salience of gender considerations. The remaining 800 participants complete these same questions after the hiring task, serving as the control group for demand effects.
At the beginning of the survey, half of the participants (800) will receive an interviewer-demand-inducing prompt to increase gender salience in the task. If they are assigned to the interviewer demand effect condition, it always comes first after completing the consent form.
These are cross-randomized, and we are interested in the interaction (which has condition sizes of approximately 400 participants.
2. Productivity Estimation Training Task
Participants are randomly assigned (800 each) to either:
With training: Complete 10 individual productivity estimation exercises before the main task
Without training: Proceed directly to the main hiring task
Among those receiving the training task, the interviewer assignment is randomized:
800 total participants evaluate candidates prior to the comparison task
400 participants evaluate candidates interviewed by Interviewer A
400 participants evaluate candidates interviewed by Interviewer B
Within each of these, participants also receive information and adjustment levels equal to their initial allocation in the comparison task above
All participants (1600) will receive information about the outcome of candidates in the preliminary productivity guessing task, even if they did not complete it
3. Feedback timing (within guessing task)
For participants in the no-adjustment condition who receive the training task, the task is randomized into two starting conditions:
Approximately 280 participants begin with a joint hiring task of five candidates before receiving feedback to test the impact of multiple delayed signals versus updating after individual hires. Simultaneously, this allows for testing whether bias is reduced or increased in simultaneous hiring tasks. Participants then select individuals one at a time and receive individual feedback from the 6th to 10th candidates.
Approximately 280 participants receive individual feedback from the beginning within the no-adjustment condition
All participants (approximately 240) in the adjustment conditions who are assigned to do the guessing task complete the individual feedback version of the task. This brings the total to 520.
4. Display Position Randomization (800 participants each)
To control for display position effects and identify whether candidate positions cause differential learning:
In the left column, 800 participants see male candidates (labeled position A, interviewer A, etc.)
In the left column, 800 participants see female candidates (labeled position A, interviewer A, etc.)
This treatment condition is purely intended as a control or counterbalance.
5. Cognitive Assessment Timing (800 participants each)
The Raven's Progressive Matrices assessment is administered either:
Before the main task for 800 participants
After the main task for 800 participants
· Participants receive the matrices either 1) just before completing the guessing task (if assigned), 2) just before completing the main task (if not assigned the guessing task), or 3) at the end of the survey. Groups 1 and 2 are combined for analysis (800 people) vs group 3 (800 people).
6. Population vs sample information shock
This is the same category discussed as a subcategory of 1 and 7 in the main treatment randomization shown above. The main purpose of this experiment is to mimic a real-world scenario where external information is used to intervene using broader population-level data. However, to partially separate the impact of improved information and simply information salience, 60 participants in each of the gender known and gender unknown conditions receive a salience shock regarding information already visible to them through the past 10 hires.
Primary outcomes
There are three families of primary outcomes, although the third set is of distinctly lower importance. The (a) tasks are of the highest importance. The (b) tasks are still within the primary outcome family but are somewhat less central (but still of higher importance than the outcomes listed in secondary outcomes). In cases where (b) uses data different from (a) within a family (e.g., the comparison vs guessing tasks), the family-wise adjustment will be done separately on sets a and b.
Family 1: Efficiency
1a) For each of the 40 decisions in the main comparison task:
1) Did the participant correctly choose the candidate with higher productivity? (“CorrectDecision”)
2) What was the productivity of the chosen candidate? (“SelectedProductivity”)
3) Did the candidate choose what a Bayesian would (or a similar optimal decision rule) (“BayesianMatch”)
1b) For each of the 10 productivity guesses in the pre-task:
1) How far was the guess from the actual productivity? (“GuessDiff”)
2) How far was the guess from the optimal guess based on prior knowledge? (“BayesianGuessDiff”)
Family 2: Equity
Note: 2a)-1 Is the primary outcome of focus for this family and will be considered separately before evaluating it jointly with outcomes 2 and 3. For each of the 40 decisions in the main comparison task:
2a)
1) Did the participant choose a man or a woman to hire? (“SelectedMan”)
2) Did this differ from the optimal gender to hire in a particular comparison? (Separated into “SelectedManCorrect”,”SelectedWomanCorrect”)
3) Did the participant’s choice of man or woman differ from a Bayesian choice? (SelectedManBayesian;SelectedWomanBayesian)
2b) For each of the 10 productivity guesses in the pre-task:
Did participants assigned to women systematically underestimate their ability? (WomenScoreDiffSigned)
Did participants assigned to men systematically overestimate their ability? (ManScoreDiffSigned)
Family 3: Attitudes and non-task Performance
All “outcomes” here have a dual purpose as inputs into the above regressions.
3a)
How well did the participant do on a 4 Raven’s Matrices subtest? (Raven-4-score)
How sexist were participant responses on the Old-fashioned Sexism scale?
How sexist were participant responses on a combined set of other sexism responses?
3b)
Participant responses to Internal-External Locus of Control Scale
Two Just World scale items and associated measures
Secondary outcomes
As a secondary analysis, I will take each individual-task-level outcome and evaluate the clustered units as a single outcome to provide an alternative approach to assess learning over time. I will also examine whether variables intended as pure controls (e.g., exposure to women) are shaped by treatment conditions. If they are, failing to control these outcome shifts could bias the primary endpoints discussed above. Specific secondary outcomes of interest include evaluating the first five or 10 comparison decisions as a unit and evaluating simple treatment effects without incorporating learning over time.
Other secondary outcomes of interest include the speed at which participants complete the Raven’s matrices tasks (especially if getting them right).
Outside of secondary outcomes, it is essential to note that there are secondary analyses (or planned supplemental analyses) of the primary outcomes listed above. While not explicitly defined as an outcome, it is essential to note that in many cases, a central estimand of interest is learning (or learning speed). This is identified by evaluating the extent to which productivity, equity, and decision-making improve across position-numbers (or lack thereof) and how this speed compares with a Bayesian. In addition, many analyses will focus on how treatment shifts productivity and equity levels versus slopes.
Cited by
No citation data available for this registration yet.