Designing job recommendation algorithms for job seekers: trading-off suitability to search criteria and hiring chances

How to cite

Perennes, Elia; Bied, Guillaume; Caillou, Philippe; Gaillac, Christophe; Sebag, Michèle; Crépon, Bruno. 2022. "Designing job recommendation algorithms for job seekers: trading-off suitability to search criteria and hiring chances." AEA RCT Registry. AEARCTR-0008998. https://doi.org/10.1257/rct.8998-1.3

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Elia Perennes* , Guillaume Bied , Philippe Caillou , Christophe Gaillac , Michèle Sebag , Bruno Crépon

* Primary investigator

Registry ID
AEARCTR-0008998
DOI
https://doi.org/10.1257/rct.8998-1.3
First registered
Last updated
Status
In development
Trial dates
2022-02-28 – 2022-08-31
Countries
Private
JEL codes
L86, C88, J64, J68
Registry record
http://www.socialscienceregistry.org/trials/8998

Abstract

We study the implementation of a job recommender system dedicated to the jobseekers at the French Public Employment Service (PES). For each job seeker, the system ranks vacancies according to the job seeker's chances of being hired on them. It is based on a state of the art machine learning (ML) model, based on the very rich data available at the French PES (including textual data and past hires).

The goal of this study is to implement a large scale beta test of this recommender system. We will set up a randomized controlled trial in which we will expose a set of job seekers to algorithmic recommendations of vacancies and ask them to rate the recommended job ads.

This experiment will allow us to identify, thanks to job seekers' ratings and applications, the best combination of two vacancy rankings : one of the rankings is based on an expert system that recommends matches based on the fit between the job seekers' search criteria and the characteristics of job postings ; the other one is obtained from the ML model mentionned above. We will also learn about the impact of providing job seekers with information about the ranks of recommended job ads within these two rankings.

Intervention

We study the implementation of a job recommender system dedicated to the jobseekers at the French Public Employment Service (PES). For each job seeker, this algorithm ranks the available job ads according to the job seeker's chances of being hired on each of them, using a state of the art maching learning algorithm trained on data on past hires recorded by the PES.

The purpose of this experiment is to conduct a large scale beta test of this recommender system. The goal of our intervention is to find an optimal combination between two job vacancy rankings: a ranking based on the chances of a hire taking place and another based on the suitability of the job to the jobseeker's explicit search criteria. The first ranking, called Ranking_ML (ML for machine learning), is obtained from the aforementioned recommender system. The latter is inspired by the matching score currently used at the French PES, which is computed as the weighted sum of the intensities of alignment between the job seeker's search criteria and the job posting's characteristics and requirements (e.g. job seeker's experience vs. required experience, salary sought vs. proposed salary, etc.). We call it Ranking_S (S for matching Score). This intervention will also allow us to investigate the impact of providing jobseekers with information about the ranks of recommended job ads within these two rankings.

This intervention proceeds as follows:
- Jobseekers receive an invitation email
- By clicking on a link in the email, jobseekers access an online survey
- Jobseekers are first exposed to two recommended vacancies. They are asked to evaluate: the overall relevance of the recommendation, their perception of their own chances of recruitment and their assessment of the fit with their search criteria.
- After having rated these two recommended vacancies, the jobseekers access a final page including the 2 previous ads, as well as 8 additional recommendations. They do not have to rate these 8 additional vacancies. On this final page, they have the possibility to click on recommended job ads in order to view further details and apply.

Experimental design

Job seekers are divided into 10 groups of equal size. How recommendations are computed as well as whether or not additional information is displayed will vary between these groups.
* For all of the 10 groups, recommendations are calculated by mixing the Ranking_ML and Ranking_S. These 10 groups are obtained by breaking each of 5 base groups into 2 further subgroups, as follows:
** The five base groups are recommended vacancies using a ranking algorithm based on a mixture of the Ranking_ML and Ranking_S rankings. Each group is characterised by different mixture weights.
** Each of these 5 groups is further split into two groups, which differ in terms of displayed information about the rankings of the recommended vacancies. One group is given information about the vacancies' ranks in both the Ranking_ML and Ranking_S rankings; the other group does not receive this information.

Design details

Randomization method
Randomization done in office by a computer
Randomization unit
Individual (jobseeker)
Clusters
100 000 jobseekers
Observations
100 000 jobseekers
Arms
10 000 jobseekers per treatment group
Primary outcomes
Collected during and following the beta test: - ratings given by jobseekers to recommended job ads. - clicks of jobseekers on recommended job ads. - applications of jobseekers to recommended job ads. Impacts on: - job search behavior: quantity and timing of search effort and the characteristics of jobs applied to. - general labor market outcomes : unemployment duration and hiring (contract types -temporary or long term-, geographic and sectoral mobility)

Cited by

No citation data available for this registration yet.

Keywords