Impact of AI Skills on Callback in Job Applications: Evidence from a Field Experiment

How to cite

Niemann, Lukas; Danilov, Anastasia; Firpo, Teo. 2021. "Impact of AI Skills on Callback in Job Applications: Evidence from a Field Experiment." AEA RCT Registry. AEARCTR-0007097. https://doi.org/10.1257/rct.7097-1.2000000000000002

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Lukas Niemann* , Anastasia Danilov , Teo Firpo

* Primary investigator

Registry ID
AEARCTR-0007097
DOI
https://doi.org/10.1257/rct.7097-1.2000000000000002
First registered
Last updated
Status
On going
Trial dates
2021-03-02 – 2021-05-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/7097

Abstract

We investigate the demand of AI skills in real job applications with a correspondence experiment. We send fictitious job applications in response to real job postings for entry-level employees with a business administration background on UK-based online job boards, randomly varying whether the CVs include AI skills. First, we study the effect of these skills on the callback rate. Second, we investigate potential differences across three types of business administration jobs (marketing, HR and finance and accounting).

Intervention

We adopt a 3 x 2 design. For each of the three types of business administration jobs (marketing, HR and finance and accounting) we create a CV with cover letter. The documents across the three different job types are comparable regarding skills and experience. Then, we randomly add AI skills to each applicant’s documents or not. Apart from AI skills, the applications within each job type are exactly the same. We then select suitable job postings from UK-based online job boards, in each of the three job categories. We send CVs and cover letters for the specific job type, randomizing whether the CV and cover letter include AI skills.

Experimental design

We use a 3 x 2 experimental design to investigate the effect of AI skills on job search success. First, we create a control CV without AI capabilities including a motivation letter for a fictitious applicant for each job type (marketing, HR and finance and accounting). In a second variant, we add additional AI skills to the applicant profile. In summary, we have two CVs for each of three different job types. For each job type, we search for matching job postings for entry-level employees with a business administration background on UK-based online job boards and randomly send (with vs without AI skills) an applicant profile to the employer. Conditional on being able to obtain the UK company number of the respective company we also collect data on the SIC code and age of the company (using the UK’s Companies House database). We track callback from employers for each observation. We first compare the callback rate (for both, sensu stricto and sensu lato) for all applications including AI skills (across all three job types) to all applications without AI skills. We then compare the callback rate by job type and presence of AI skills.

Design details

Randomization method
The randomization is stratified by job type. Because we cannot control how many job postings for each job type will be available on any given week, and we would like to ensure numerical balance between the treatment and control groups over the timeline, we randomize job postings in blocks of four (e.g., Treatment-Control-Control-Treatment). Between March and April 2021, we will search for suitable job postings in each job type on a weekly basis and apply. We also randomize the order of job types in which we search for, and apply to, job postings (such that we do not always start with e.g., ‘marketing’ and end with ‘HR’, but rather rotate through them on a daily basis). To maintain the integrity of the randomization, we will respect the block nature of the randomization sequence (i.e., ensuring there are at least four applications available before starting with a new block). We will only send applications to job postings that do not immediately require a test or supplementary information (beyond a CV and cover letter), to avoid immediate rejections due to insufficient information. We will not send more than one application to a company, to avoid detection. Randomization was carried out in Stata (seed 12345); randomization code is available upon request.
Randomization unit
Job posting
Clusters
NA
Observations
600
Arms
300 applications with treatment and 300 applications control, equally split between the three job types (such that each trial arm has 100 applications).
Primary outcomes
Our primary outcome is callback from the employer. In general, callback is a response from the employer indicating interest in the applicant. We differentiate between two different versions of callback as in Baert et al. (2013). 1. Callback sensu stricto is every callback from an employer that implies a direct job offer or an invitation to an interview. Not considered are callbacks that imply any other positive response, rejections, and no responses. 2. Callback sensu lato includes every callback from an employer that implies any positive response. Not considered are rejections and no responses. Callbacks are received by email or voice mail. References Baert, S., Cockx, B., Gheyle, N. & Vandamme, C. (2013). Do Employers Discriminate Less if Vacancies are Difficult to Fill? Evidence from a Field Experiment. CESifo Working Paper Series 4093, CESifo.
Secondary outcomes
Additionally, we measure the response type. This is an ordinal variable and possible values in descending order are: direct job offer, invitation to an interview, invitation to an interview for an alternative position, phone voice mail with request to call back, invitation to an assessment center, phone call without voice mail message, request for additional information, rejection, no response. We also measure the response time. This is defined as the time between sending the application and receiving an answer by email or voice mail from the employer.

Cited by

No citation data available for this registration yet.

Keywords