Cover Letter Evaluations and LLMs

How to cite

Wicker, Till; Zaccaria, Niccolo; Musillo, Giuseppe; Nejad, Kian Abbas. 2024. "Cover Letter Evaluations and LLMs." AEA RCT Registry. AEARCTR-0014784. https://doi.org/10.1257/rct.14784-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Till Wicker* , Niccolo Zaccaria , Giuseppe Musillo , Kian Abbas Nejad

* Primary investigator

Registry ID
AEARCTR-0014784
DOI
https://doi.org/10.1257/rct.14784-1.0
First registered
Last updated
Status
In development
Trial dates
2024-11-11 – 2024-11-29
Countries
Private
JEL codes
J08, J64, C90
Registry record
http://www.socialscienceregistry.org/trials/14784

Abstract

The use of Large Language Models (LLM) could have a profound impact on the labor market. The first part of this research project (AEARCTR-0013355) illustrates that the quality of cover letters improves as job seekers can use LLMs (such as ChatGPT) when writing their cover letter – as evaluated by recruiters who were blind to the treatment arms. However, the second part of the research project focuses on recruiters, to understand their perceptions of the use of LLMs in cover letters and job applications more generally. This is done through an online lab experiment, with recruiters from Prolific.

Intervention

Recruiters are randomly assigned to control group (C), partial info (T1), or full info (T2). All recruiters will be shown five cover letters that they have to evaluate based on a pre-defined evaluation criteria. These five cover letters are random drawn from a sample of 12 cover letters, written by job seekers who participated in AEARCTR-0013355. All cover letters refer to the same job description, which recruiters are shown. Half of the cover letters were written with the assistance of ChatGPT, and half were not. Recruiters in the C group are not informed about this possibility, while recruiters in T1 are informed that cover letters may be written with the use of ChatGPT. Recruiters in T2 get told exactly which cover letters are written with the use of ChatGPT.

Experimental design

Recruiters will be randomized across three treatments, see above. Recruiters will first see the job description, as well as evaluation criteria to be used, before being shown a cover letter. They are shown one cover letter at a time, and asked to evaluate it based on the pre-specified rubric. After evaluating five cover letters, recruiters will be asked general questions about cover letters, recruitment, and the use of LLMs.

Design details

Randomization method
Randomization done by Qualtrics, without stratification.
Randomization unit
Individual
Clusters
400 individuals.
Observations
400 individuals.
Arms
400 divided across 3 treatments, equaling 133. However, every participant will evaluate multiple cover letters.
Primary outcomes
Evaluation of quality of cover letter and CV, based on a pre-specified and standardized grading criteria.
Secondary outcomes
Time spent evaluating the cover letter. Responses to post-experimental questionnaire.

Cited by

No citation data available for this registration yet.

Keywords