The Impact of AI-assistance on Recruitment Judgment

How to cite

Qiu, Jingyi; Wang, Qingyi. 2024. "The Impact of AI-assistance on Recruitment Judgment." AEA RCT Registry. AEARCTR-0013527. https://doi.org/10.1257/rct.13527-1.3

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jingyi Qiu* , Qingyi Wang

* Primary investigator

Registry ID
AEARCTR-0013527
DOI
https://doi.org/10.1257/rct.13527-1.3
First registered
Last updated
Status
In development
Trial dates
2024-04-29 – 2026-05-01
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/13527

Abstract

The increasing prevalence of artificial intelligence (AI) in the job application process raises questions about its influence on recruiters' judgments and decisions. This study investigates how AI assistance in application materials affects recruiters' judgments of job applicants, focusing on both the perceived and actual use of AI. To address this, we will conduct a field experiment involving a campus-wide job opening for research assistants. Recruiters, as the primary participants, will evaluate applicants after reviewing their resumes and cover letters. The study employs a 2x2 experimental design, manipulating both the perceived and actual use of AI in generating the cover letters. By examining how recruiters evaluate applicants under these different conditions, we seek to understand the impact of AI assistance on their recruitment judgments. Our findings will contribute to the growing body of literature on the role of AI in hiring processes and its potential implications for both applicants and organizations.

Intervention

We employ a 2×2 experimental design that manipulates both the perceived and actual use of AI-generated cover letters

Experimental design

We will conduct a field experiment in three steps. First, we will create a research assistant (RAs) position and post the job advertisement campus-wide at the University of Michigan. Applicants materials submitted will serve as the basis for evaluation. Second, we will recruit two groups of recruiters to evaluate the applicants and make hiring decisions. Our study involves two types of evaluators: PhD students with direct experience hiring RAs for similar roles, and HR professionals from Prolific. Third, each recruiter will review a series of applicant profiles, each including a resume and a cover letter. Here we employ a 2x2 experimental design.

Design details

Randomization method
Randomization done by a Qualtrics embedded randomizer
Randomization unit
individual
Clusters
NA
Observations
200 Prolific recruiters each reviewing 6 profiles 50 PhD student recruiters each reviewing 6 profiles
Arms
The randomization will be done at individual evaluation level. However, given our randomization method, the expected number of observations falling in different conditions will be: (1) 562 AI-generated and identified (2) 188 AI-generated but not detected (3) 562 human-generated and recognized (4) 188 human-generated but mistaken as AI-generated
Primary outcomes
A recruiter's overall evaluation of an applicant profile
Secondary outcomes
A recruiter's overall evaluation of an applicant's technical ability and willingness to exert effort

Cited by

No citation data available for this registration yet.

Keywords