Job Applications and LLMs
How to cite
Wicker, Till; Zaccaria, Niccolo; Musillo, Giuseppe; Nejad, Kian Abbas. 2024. "Job Applications and LLMs." AEA RCT Registry. AEARCTR-0013355. https://doi.org/10.1257/rct.13355-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
The use of Large Language Models (LLM) could have a profound impact on the labor market. There is already extensive research on the impact of AI automation on existing jobs, but little is known about the use of LLMs and AI tools on the selection process leading to new hires. For entry level positions, applicants are usually required to submit a CV and a cover letter in order to send a signal to the new employer about their skills. Writing a good cover letter that impresses an employer requires time and effort, and was traditionally interpreted as an effective way to send a signal about applicants’ relevant skills and motivation for the job. However, with the emergence of LLMs, writing a good quality cover letter has become much easier, and hence cover letters are a less reliable signal of an applicant’s quality. This is further confounded by the insights from the academic literature on the productivity effects of LLMs, which find that LLMs help everyone, however help the low-performers disproportionately more (Noy and Zhang, 2023; Dell’Acqua et al., 2023). Therefore, the difference in quality of cover letters between good and less good applicants is likely to decrease as a result of the use of LLMs – making cover letters a less reliable signal of the applicant’s quality.
Intervention
Students are randomly assigned to control group (C), treatment 1 (T1), or treatment 2 (T2). All groups undergo a training session, after that they will write a cover letter, for a hypothetical job description of one of six participating firms, based on their preference. The C group receives a generic, unrelated training and is blocked from accessing LLMs when they write a cover letter. T1 receives the same training as C but they are allowed to use LLMs to write their cover letter. T2 receives a training on how to write effective LLM prompts, before being able to use LLMs to write their cover letter.
Experimental design
Participants will be randomized across three treatments, see above. Participants will first receive a short training, before writing a cover letter to a hypothetical position at a firm of their choosing (among the six partner firms in this study). Cover letters and CVs will be evaluated by recruiters from the six firms who are blind to the interventions and experiment.
Design details
Cited by
No citation data available for this registration yet.