Understanding How Information about AI affects Response to AI in the Recruitment Process
How to cite
Avery, Mallory; Leibbrandt, Andreas.
2023. "Understanding How Information about AI affects Response to AI in the Recruitment Process." AEA RCT Registry. AEARCTR-0012314.
https://doi.org/10.1257/rct.12314-1.0
Download:
BibTeX
·
RIS— for reference managers without a browser
translator. Zotero and most
others pick up this page's citation metadata directly.
In this project, we study how providing information about AI to candidates and evaluators for a real job affects their application and evaluation behavior.
Intervention
In this project, we study how providing information about AI to candidates and evaluators for a real job affects their application and evaluation behavior.
Experimental design
Our design aims to measure the impact of providing information about AI and AI-generated bias on diversity in recruitment outcomes.
Design details
Randomization method
Randomization will be carried out by a computer.
Randomization unit
The randomization unit will be the individual for all treatments.
Clusters
The clusters will be equal to the number of legitimate candidates. Based on previous experience we expect to have 1200 legitimate job applicants, but this will likely vary depending on a number of factors. A legitimate candidate is someone who resides in the US or Australia and completes the initial application form. We plan to assign 25% of the applications to each treatment. We will cross-stratify by country, gender, and racial minority status.
For stage 2, we plan to recruit enough human evaluators so that each candidate is assessed by several human evaluators. We will decide on the exact number of potential human evaluators based on the number of candidates in stage 1. The updated number can be found in Appendix 1.
Observations
For the applicant sample (stage 1), the number of observations is the same as the number of clusters. We hope to have up to 1200 observations.
For stage 2, the AI will evaluate all candidates. The candidate pool will also be evaluated by human evaluators. Each candidate will be evaluated by several human evaluators. The number of human evaluators and the number of candidates assessed by each evaluator will be added to the pre-analysis plan before the start of stage 2 (see Appendix 1).
Arms
See above. We plan to assign 25% of the sample to all treatments. Further, we will cross-stratify by gender, racial minority status, and country.
Primary outcomes
We collect the following primary outcomes:
- Proportion who starts the candidate assessment: This is defined as the number of candidates who start the candidate assessment/ The number who receive the candidate assessment. A candidate is considered to have received the assessment if they are sent an email with details on the assessment.
-Proportion who complete: This is defined as the number who complete the candidate assessment/ The number who receive the candidate assessment. A candidate is considered to have received the assessment if they are sent an email with details on the assessment. A candidate is said to have completed if they answer all questions.
-Overall assessment evaluation score: This is made up of two parts: Overall evaluation of candidates AI score: The AI will evaluate all candidates and score them on a scale between 0-100 (where 0 means low and 100 high). Overall evaluation of candidates Human: Our human evaluators will score all candidates on a scale between 0-100.
Note: Any metrics calculated by the AI algorithm such as, the candidates overall score, are set by the tech firm we are collaborating with and is not influenced by the researchers. The algorithm will not change throughout the project.
Secondary outcomes
To understand possible mechanisms we elicit the following secondary outcome variables:
Time to complete the questions: the candidates completion time is recorded.
Number of words: This outcome is the number of words written in the assessment.
Second order overall evaluation of candidates Human: Our human evaluators will also be asked to score candidates based on their expected score of the median human evaluator. The score will be on a scale between 0-100.
Cited by
No citation data available for this registration yet.