Understanding How Information about AI affects Response to AI in the Recruitment Process

How to cite

Avery, Mallory; Leibbrandt, Andreas. 2024. "Understanding How Information about AI affects Response to AI in the Recruitment Process." AEA RCT Registry. AEARCTR-0013832. https://doi.org/10.1257/rct.13832-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Mallory Avery* , Andreas Leibbrandt

* Primary investigator

Registry ID
AEARCTR-0013832
DOI
https://doi.org/10.1257/rct.13832-1.0
First registered
Last updated
Status
In development
Trial dates
2024-06-19 – 2024-07-19
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/13832

Abstract

In this project, we study how providing information about AI to candidates and evaluators for a real job affects their application and evaluation behavior.

Intervention

In this project, we study how providing information about AI to candidates and evaluators for a real job affects their application and evaluation behavior.

Experimental design

Our design aims to measure the impact of providing information about AI and AI-generated bias on diversity in recruitment outcomes.

Design details

Randomization method
Randomization will be carried out by a computer.
Randomization unit
The randomization unit will be the individual for all treatments.
Clusters
The clusters will be equal to the number of legitimate candidates. Based on previous experience we expect to have 1200 legitimate job applicants, but this will likely vary depending on a number of factors. A legitimate candidate is someone who resides in the US or Australia and completes the initial application form. We plan to assign 25% of the applications to each treatment. We will cross-stratify by country, gender, and racial minority status. For stage 2, we plan to recruit enough human evaluators so that each candidate is assessed by several human evaluators. We will decide on the exact number of potential human evaluators based on the number of candidates in stage 1. The updated number can be found in Appendix 1.
Observations
For the applicant sample (stage 1), the number of observations is the same as the number of clusters. We hope to have up to 1200 observations. For stage 2, the AI will evaluate all candidates. The candidate pool will also be evaluated by human evaluators. Each candidate will be evaluated by several human evaluators. The number of human evaluators and the number of candidates assessed by each evaluator will be added to the pre-analysis plan before the start of stage 2 (see Appendix 1).
Arms
See above. We plan to assign 25% of the sample to all treatments. Further, we will cross-stratify by gender, racial minority status, and country.
Primary outcomes
We collect the following primary outcomes: - Proportion who starts the candidate assessment: This is defined as the number of candidates who start the candidate assessment/ The number who receive the candidate assessment. A candidate is considered to have received the assessment if they are sent an email with details on the assessment. -Proportion who complete: This is defined as the number who complete the candidate assessment/ The number who receive the candidate assessment. A candidate is considered to have received the assessment if they are sent an email with details on the assessment. A candidate is said to have completed if they answer all questions. -Overall assessment evaluation score: This is made up of two parts: Overall evaluation of candidates AI score: The AI will evaluate all candidates and score them on a scale between 0-100 (where 0 means low and 100 high). Overall evaluation of candidates Human: Our human evaluators will score all candidates on a scale between 0-100. Note: Any metrics calculated by the AI algorithm such as, the candidates overall score, are set by an external AI provider and is not influenced by the researchers. The algorithm will not change throughout the project.
Secondary outcomes
To understand possible mechanisms we elicit the following secondary outcome variables: Time to complete the questions: the candidates completion time is recorded. Number of words: This outcome is the number of words written in the assessment.

Cited by

No citation data available for this registration yet.

Keywords