DESIGNING JOB RECOMMENDATION ALGORITHMS FOR JOB SEEKERS: SECOND BETA TEST
How to cite
Crepon, Bruno; Bied, Guiilaume; Caillou, Philiipe; Gaillac, Christophe; Perennes, Elia; Sebag, Michele. 2025. "DESIGNING JOB RECOMMENDATION ALGORITHMS FOR JOB SEEKERS: SECOND BETA TEST." AEA RCT Registry. AEARCTR-0016650. https://doi.org/10.1257/rct.16650-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We study the implementation of a job recommender system designed for jobseekers registered with the French Public Employment Service (PES). For each jobseeker, the system ranks vacancies according to their estimated chances of being hired. It relies on a state-of-the-art machine learning (ML) model trained on the rich administrative data available at the French PES, including textual information and past hires.
The aim of this study is to conduct a large-scale beta test of various versions of this recommender system. It follows a previous large-scale beta test [https://doi.org/10.1257/rct.8998-1.3]. We will set up a randomized controlled trial in which jobseekers are exposed to algorithmic job recommendations generated by different algorithms informed by the results of our first beta test. As in the first test, jobseekers will be asked to rate the recommended job ads.
Intervention
We study the implementation of job recommender systems for jobseekers at the French PES. For each jobseeker, an initial algorithm ranks job ads according to their estimated hiring probability, using a state-of-the-art machine learning algorithm trained on past hires recorded by the PES. We refer to this version as V.0.
Our first beta test [registered as https://doi.org/10.1257/rct.8998-1.3] compared the performance of recommendations generated by V.0 with (i) those from an algorithm based on the match between job ads and search criteria (S) and (ii) those based on a combination of the rankings from these two polar systems, V.0 and S.
This first beta test showed the value of combining information from both rankings. It led to the development of a new algorithm, V.2, which introduced several innovations (For a comprehensive exposition, see our paper: Bied, G. and al. (2023). Toward job recommendation for all. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI), AI for Good , 5906–5914.
). First, the algorithm’s architecture was modified, introducing a two-step procedure, in which the top 1000 jobs first selected by V.0 are re-ranked by a more expressive neural model in a second stage. Second, the components used in S were incorporated as predictors in the second-stage hiring model. Third, part of the second-stage model was dedicated to predict job seekers’ applications (algorithm A see below); the resulting application probability score was itself then used as a component in the prediction of hiring probability.
We also considered the algorithm (A) in isolation. This algorithm follows the same learning structure as V.0 and the second stage akin to V.2’s but with the purpose to predict applications.
The main goal of the current beta test is to compare the performance of V.0 and S with that of V.2 and A. However, we also include a new algorithm combining the rankings of V.2 and S—similar to the first beta test—referred to as Mixture.
In addition, we include an algorithm trained with XGBOOST, which represents the state of the art, while the other machine learning algorithms are trained as neural networks.
A final feature of this beta test is the inclusion of recommendations from an expert system, S*, currently used by the PES. Our version S is only an approximation. The S* algorithm also applies numerous filters, which means it does not necessarily return offers for every jobseeker.
The algorithms compared are therefore: S*, S, V.0, V.2, A, Mixture, and XGBOOST.
Additional treatment arms consider recommendations from the V.2, A and Mixture algorithm (renamed V.2.I0, A.I0 and Mixture.I0), to which varying levels of explanation are cumulatively added:
The first level focuses on the purpose of the algorithm (V.2.I1, A.I1 and Mixture.I1),
The second on its performance (V.2.I2, A.I2 and Mixture.I2),
The third on the rationale behind its relevance (V.2.I3, A.I3 and Mixture.I3).
The experiment proceeds as follows:
Jobseekers receive an invitation email.
By clicking on a link in the email, they access an online survey.
Jobseekers are first shown five recommended vacancies and asked to evaluate the overall relevance of the recommendations.
After rating these five vacancies, they reach a final page displaying the same five ads, along with five additional recommendations. They are not required to rate the five additional vacancies. On this final page, they can click on job ads to view details and apply.
Experimental design
The experiment proceeds as follows:
It took place in the Rhone-Alpes Region and considered job seekers ready to start a job immediately. 160.000 jobseekers were drawn at random from the pool of eligible jobseekers. These jobseekers were randomly assigned to 16 groups each of size 10,000. Each group was assigned to receive recommendations generated by one of the following algorithms: S*, S, V.0, V.2.I0, A.I0, Mixture.I0, V.2.I1, A.I1, Mixture.I1, V.2.I2, A.I2, Mixture.I2, V.2.I3, A.I3, Mixture.I3, and XGBOOST.
Jobseekers receive an invitation email.
Each jobseeker receives the same invitation email, regardless of their assigned group. They are considered enrolled in the experiment from the moment they click on the link.
By clicking on a link in the email, they access an online survey.
Jobseekers are first shown five recommended vacancies and asked to evaluate the overall relevance of the recommendations.
After rating these five vacancies, they reach a final page displaying the same five ads, along with five additional recommendations. They are not required to rate the five additional vacancies. On this final page, they can click on job ads to view details and apply.
Design details
Cited by
No citation data available for this registration yet.