Cover Letter Evaluations and LLMs
How to cite
Wicker, Till; Zaccaria, Niccolo; Musillo, Giuseppe; Nejad, Kian Abbas. 2024. "Cover Letter Evaluations and LLMs." AEA RCT Registry. AEARCTR-0014784. https://doi.org/10.1257/rct.14784-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
The use of Large Language Models (LLM) could have a profound impact on the labor market. The first part of this research project (AEARCTR-0013355) illustrates that the quality of cover letters improves as job seekers can use LLMs (such as ChatGPT) when writing their cover letter – as evaluated by recruiters who were blind to the treatment arms. However, the second part of the research project focuses on recruiters, to understand their perceptions of the use of LLMs in cover letters and job applications more generally. This is done through an online lab experiment, with recruiters from Prolific.
Intervention
Recruiters are randomly assigned to control group (C), partial info (T1), or full info (T2). All recruiters will be shown five cover letters that they have to evaluate based on a pre-defined evaluation criteria. These five cover letters are random drawn from a sample of 12 cover letters, written by job seekers who participated in AEARCTR-0013355. All cover letters refer to the same job description, which recruiters are shown. Half of the cover letters were written with the assistance of ChatGPT, and half were not. Recruiters in the C group are not informed about this possibility, while recruiters in T1 are informed that cover letters may be written with the use of ChatGPT. Recruiters in T2 get told exactly which cover letters are written with the use of ChatGPT.
Experimental design
Recruiters will be randomized across three treatments, see above. Recruiters will first see the job description, as well as evaluation criteria to be used, before being shown a cover letter. They are shown one cover letter at a time, and asked to evaluate it based on the pre-specified rubric. After evaluating five cover letters, recruiters will be asked general questions about cover letters, recruitment, and the use of LLMs.
Design details
Cited by
No citation data available for this registration yet.