Recruitment Screening with Human and Artificial Intelligence

How to cite

Yanagizawa-Drott, David; Krenk, Ursa; Awuah, Kobbina. 2023. "Recruitment Screening with Human and Artificial Intelligence." AEA RCT Registry. AEARCTR-0011651. https://doi.org/10.1257/rct.11651-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

David Yanagizawa-Drott* , Ursa Krenk , Kobbina Awuah

* Primary investigator

Registry ID
AEARCTR-0011651
DOI
https://doi.org/10.1257/rct.11651-1.1
First registered
Last updated
Status
Completed
Trial dates
2023-07-05 – 2023-08-30
Countries
Ghana ()
Registry record
http://www.socialscienceregistry.org/trials/11651

Abstract

This study aims to investigate the potential role of artificial intelligence (AI) in assisting in the screening for labor in the recruitment process, with the use of generative large language models.

We will conduct a randomized controlled trial (RCT) in assisting a non-profit organization in recruiting talent – teachers – within the education sector in a sub-Saharan country. The experiment will involve human evaluators who assess incoming applications for the NGO's fellowship program, with or without the assistance of AI. The human evaluators are tasked to screen applications and decide who continues to the next in-depth interview phase, based on a set of criteria deemed desirable to the organization. We give the same task to the generative AI using prompts. We will explore the disagreement between human and AI in evaluating applications, the human behavioral responses under the AI-assistance in terms of effort and application grading, and the quality of candidates among those that are passed onto the interview phase.

Applications will be randomized into four treatment groups: Human-only (control); Human with AI score assistance; Human with AI score and rationale assistance, and; AI-only. The treatment group will determine whether the evaluator will receive AI assistance in scoring an application and what score (AI-generated or evaluator-determined) will be used to select applicants. In addition to behavioral responses by human evaluators when assisted, we will compare the efficiency of AI-only and AI-assisted recruitment to human-only screening, focusing on the quality of selected applicants, screening costs, and speed.

Intervention

The intervention consists in providing evaluators - people who assess incoming applications - with AI assistance. AI assistance can take two forms: providing them with the scores that the AI would have predicted for a given application, or providing them with the scores and an AI-generated explanation for the choice. The AI-only condition assesses applications without human grading and decisions.

Experimental design

All grading will take place on the Qualtrics platform. We randomize applications into four treatment groups: Human-only decisions (control); Human decisions with AI score assistance; Human decisions with AI score and rationale assistance, and; AI-only decisions. Thus, the treatment group will determine two things: (1) whether the evaluator will receive AI assistance in scoring an application and (2) what score (AI-generated or evaluator-determined) will be used to select applicants. The human evaluators will be informed whether and what type of AI assistance will be given, at the beginning of an application. If assistance is given, it will be provided after the initial score is entered by the human evaluator. Once entered, the information from the AI assistance will be revealed and the human evaluator will be asked for a final score.

Design details

Randomization method
Randomization done in office by a computer.
Randomization unit
Application.
Clusters
We do not know in advance how many applications there will be. The hope is that at least 400 applications will be received.
Observations
For each application, we will observe grading on 6 questions. We do not know in advance how many applications there will be. The hope is that at least 400 applications will be received, in which case 2400 application-question scores will be the number of observations in total across all four conditions. In the subset of human-only and human with AI-assistance, it would result in 1800 application-question scores. For application outcomes measured in the assessment center, we do not know in advance the number of observations, as it depends on how many will be receiving a passing grade.
Arms
We do not know in advance how many applications there will be. The hope is that at least 400 applications will be received, in which case we will have 100 applications across the four conditions.
Primary outcomes
Using data from the first three conditions, we can estimate the behavioral responses by human evaluators under AI assistance, in terms of: Time spent grading the applications (effort) Grading decisions (a dummy for whether the AI and human evaluators’ final scores are the same; a continuous measure of disagreement in scores) For application outcomes, we will examine the quality of the top-k selected candidates in each randomized group, as measured in the NGO’s in-depth interviews during the assessment center phase.

Cited by

No citation data available for this registration yet.

Keywords