Understanding the Implications of AI in the Job Application Process.

How to cite

Vecci, Joe; Leibbrandt, Andreas; Avery, Mallory; Adamovic, Mladen. 2021. "Understanding the Implications of AI in the Job Application Process.." AEA RCT Registry. AEARCTR-0008296. https://doi.org/10.1257/rct.8296-1.2

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Joe Vecci* , Andreas Leibbrandt , Mallory Avery , Mladen Adamovic

* Primary investigator

Registry ID
AEARCTR-0008296
DOI
https://doi.org/10.1257/rct.8296-1.2
First registered
Last updated
Status
Completed
Trial dates
2021-09-29 – 2021-10-30
Countries
United States of America ()
Registry record
http://www.socialscienceregistry.org/trials/8296

Abstract

In this project, we study whether candidates for a real job behave differently when they are informed their application will be evaluated by AI relative to a human evaluator. We also study the differences in the characteristics of candidates recommended by AI relative to a human evaluator and perceptions of AI recruitment tools. Finally, we attempt to understand any potential differences between AI and human evaluations.

Intervention

In the project we study whether candidates for a real job behave differently when their applications are evaluated by AI relative to a human evaluator.

Experimental design

Our design aims to measure the impact of AI relative to human evaluation of job candidates.

Design details

Randomization method
Randomization will be carried out by a computer.
Randomization unit
The randomization unit will be the individual for all treatments.
Clusters
The clusters will be equal to the number of legitimate candidates. Based on previous experience we expect to have 1200 legitimate job applicants, but this will likely vary depending on a number of factors. A legitimate candidate is someone who resides in the United States and completes the initial application form. We plan to assign 30% of the sample to the Inform AI treatment and 70% to the Inform Human evaluation treatment. Further, we will stratify by gender such that there is the same proportion of men/women in each information treatment. For stage 2, we plan to recruit enough human evaluators so that each candidate is assessed by several human evaluators. We will decide on the exact number of potential human evaluators based on the number of candidates in stage 1. The updated number can be found in Appendix 1.
Observations
For the applicant sample (stage 1), the number of observations is the same as the number of clusters. We hope to have up to 1200 observations. For stage 2, the AI will evaluate all candidates. The candidate pool will also be evaluated by human evaluators each candidate will be evaluated by several human evaluators. The number of human evaluators and the number of candidates assessed by each evaluator will be added to the pre-analysis plan before the start of stage 2 (see Appendix 1).
Arms
See above. We plan to assign 30% of the sample to the Inform AI treatment and 70% to the Inform Human evaluation treatment. Further, we will stratify our sample by gender such that 70% of the overall male and female sample will be assigned to the Inform Human evaluation treatment and 30% to the Inform AI treatment.
Primary outcomes
We collect the following primary outcomes: - Proportion who starts the candidate assessment: This is defined as the number of candidates who start the candidate assessment/ The number who receive the candidate assessment. A candidate is considered to have received the assessment if they are sent an email with details on the assessment. -Proportion who complete: This is defined as the number who complete the candidate assessment/ The number who receive the candidate assessment. A candidate is considered to have received the assessment if they are sent an email with details on the assessment. A candidate is said to have completed if they answer all questions. -Overall assessment evaluation score: This is made up of two parts: Overall evaluation of candidates AI score: The AI will evaluate all candidates and score them on a scale between 0-100 (where 0 means low and 100 high). Overall evaluation of candidates Human: Our human evaluators will score all candidates on a scale between 0-100. Note: Any metrics calculated by the AI algorithm such as, the candidates overall score, are set by the tech firm we are collaborating with and is not influenced by the researchers. The algorithm will not change throughout the project.
Secondary outcomes
To understand possible mechanisms we elicit the following secondary outcome variables: -Belief about bias: Candidates are asked to rate their expected level of bias when a human evaluates their assessment and also when an AI evaluates their assessment. In this analysis we compare these two responses. -Belief about fairness: Candidates are asked to rate their expected and experienced level of fairness when a human evaluates their assessment and also when a AI evaluates their assessment. In this analysis we compare these two responses. -Overall satisfaction with assessment process: All candidates are asked to rate the assessment process on a scale between 1 and 10 (where 10 represented most satisfied). Beliefs about value of applicant in organization: All candidates are asked to rate whether the application process made them feel valued by the company. Beliefs about status of applicant in organization: All candidates are asked to rate whether they believe they will have high or low status within the company. Time to complete the questions: the candidates completion time is recorded. Number of words: This outcome is the number of words written in the assessment. Second order overall evaluation of candidates Human: Our human evaluators will also be asked to score candidates based on their expected score of the median human evaluator. The score will be on a scale between 0-100. -We also ask human evaluators the rationale for their overall evaluation score. We use this to understand potential mechanisms. Heterogeneity Analysis : We will also use a number of variables to understand the influence of AI relative to humans on candidate diversity. Our main focus will be on gender but as a secondary analysis conditional on obtaining sufficient observations we also study ethnicity. To focus on diversity, we will interact the treatment variables by the gender and possibly ethnicity (separately) of the candidate. We will also include a variable indicating whether the evaluator is HR or a Web Designer. We also conduct a number of robustness tests: -We use the Double Lasso method to select possible control variables. -To understand whether the genderness of the assessment text influences the human and AI evaluation score we will use an AI algorithm to rate the probability the assessment text is written by a male or female. - Experimenter demand effect may influence the behavior of human evaluators (i.e., human evaluators may guess the aim of the project/or because they are observed they may inflate the scores of minorities/females to avoid being perceived as behaving in a socially undesirable manner). To measure this, we ask human evaluators what they think is the aim of the study. -As a robustness to completion and application rates, we also track the proportion of people who open the email.

Cited by

No citation data available for this registration yet.

Keywords