Aversion to job recommendation algorithms among job seekers: a field experiment to measure it and understand what cause it
How to cite
Perennes, Elia; Bied, Guillaume. 2023. "Aversion to job recommendation algorithms among job seekers: a field experiment to measure it and understand what cause it." AEA RCT Registry. AEARCTR-0011157. https://doi.org/10.1257/rct.11157-1.2
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We study the implementation of a job recommender system dedicated to the jobseekers at the French Public Employment Service (PES). The recommender system is a combination of two vacancy rankings : the first ranking is based on an expert system that recommends matches based on the fit between the job seekers' search criteria and the characteristics of job postings ; the other one is obtained from a state of the art machine learning (ML) model, based on the very rich data available at the French PES (including textual data and past hires).
The objectives of this study are threefold: first, it will measure the degree of aversion (and/or acceptance) of job seekers for algorithmic recommendations ; second, it will determine the mechanisms that cause aversion ; and third, it will investigate how the framing of recommendations can minimize the risk of aversion.
We set up a randomized controlled trial in which a set of job seekers is exposed to algorithmic recommendations of vacancies. Several framing alternatives are tested: the first explains to job seekers the global functioning of the algorithm, the second emphasizes the level of performance of the algorithm and the third emphasizes the fact that the algorithm has been co-constructed with job seekers. During the experiment, job seekers are explicitly asked about how they perceive the recommendations. I also use implicit satisfaction measures, such as the click rate on the recommended ads.
Intervention
The intervention consists in exposing a set of job seekers to algorithmic recommendations of vacancies. The recommender system used in the intervention is a combination of two vacancy rankings : a ranking based on the chances of a hire taking place and another based on the suitability of the job to the jobseeker's explicit search criteria. The first ranking is obtained from a state of the art maching learning algorithm trained on data on past hires recorded by the PES. The latter is inspired by the matching score currently used at the French PES, which is computed as the weighted sum of the degrees of adequacy between the job seeker's search criteria and the job posting's characteristics and requirements (e.g. job seeker's experience vs. required experience, salary sought vs. proposed salary, etc.).
The intervention proceeds as follows:
- Selected jobseekers receive an invitation email.
- By clicking on a link in the email, jobseekers access an online survey.
- Job seekers are first exposed (or not depending on the treatment group, see below) to an infographic giving details on the algorithm that generated the recommendations.
- Job seekers see 5 vacancies, which are recommended to them by the algorithm.
- Job seekers can (if they want to) give their opinion on each recommendation by indicating whether or not the vacancy interests them.
- Job seekers are then asked three questions regarding: their level of understanding of the process behind the recommendations, their chances of being hired on recommendations, and their perception that the recommendations are based on job search experience of previous job seekers.
- Once the job seekers have answered these three questions, a new page with the 5 recommended ads (those viewed just before) is displayed. On this page, they have the possibility to click on recommended job ads in order to view further details and apply.
Experimental design
Job seekers are divided into 6 groups of equal size. Two elements vary between these groups: (1) the perceived source of the recommendations (3 possible conditions) and (2) the display of additional information accompanying the recommendations (4 possible conditions).
The three alternatives regarding the perceived source of recommendations are:
- "Machine source": Recommendations are framed as coming from an artificial intelligence algorithm
- "Human source": Recommendations are framed as coming from a (human) labor market expert
- "Neutral source": Recommendations are not framed as coming from a particular source
The four alternatives regarding the recommendations framing are:
- "Global explanation": explains to job seekers the overall functioning of the algorithm
- "Performance": emphasizes the performance level of the algorithm
- "Human in the loop" : highlights the fact that the algorithm was co-constructed with job seekers
- "Control": no particular framing
The 6 treatment groups are (Source x Framing) :
* 1: Machine source x Control
* 2: Machine source x Global explanation
* 3: Machine source x Performance
* 4: Machine source x Human in the loop
* 5: Human source x Control
* 6: Neutral source x Control
*** Preview links:
Treatment 2: https://www.dropbox.com/s/5ut0uh0gwuf9jf7/Projet%20aversion%20-%20version%20explication%20globale.mov?dl=0
Treatment 3: https://www.dropbox.com/s/uemfvw4dm37lrng/Projet%20aversion%20-%20version%20performance.mov?dl=0
Treatment 4: https://www.dropbox.com/s/j2rro7efgk0kmte/Projet%20aversion%20-%20version%20feedback.mov?dl=0
Treatment 5: https://www.dropbox.com/s/3qfwaqfmpazits1/Projet%20aversion%20-%20version%20expert.mov?dl=0
Treatment 6: https://www.dropbox.com/s/8lchw0478j1rkl3/Projet%20aversion%20-%20version%20neutre.mov?dl=0
(Treatment 1: no preview available, but this treatment corresponds to treatment 2 (or 3 or 4) without the "framing" part).
Design details
Cited by
No citation data available for this registration yet.