Increasing diversity: Experimental evidence

How to cite

Thon, Max; Fuchs, Larissa; Heinz, Matthias; Pinger, Pia. 2022. "Increasing diversity: Experimental evidence." AEA RCT Registry. AEARCTR-0010433. https://doi.org/10.1257/rct.10433-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Max Thon* , Larissa Fuchs , Matthias Heinz , Pia Pinger

* Primary investigator

Registry ID
AEARCTR-0010433
DOI
https://doi.org/10.1257/rct.10433-2.0
First registered
Last updated
Status
On going
Trial dates
2022-10-18 – 2024-04-30
Countries
Private
JEL codes
M51, M52, D22
Registry record
http://www.socialscienceregistry.org/trials/10433

Abstract

In this project, we aim to study the ability for organizations to increase diversity through job advertisements that highlight certain job characteristics, namely job flexibility and wage growth. We further investigate how such changes to job advertisements affect the perceived attractiveness of a position and applicants’ expectations in terms of working conditions, working environment, and wages. More generally, we study how the highlighting of certain job characteristics in job ads changes the composition of the applicant pool.

We conduct an RCT and a survey experiment. In this project, we cooperate with a large tech firm. In an RCT, we vary the design of job advertisements to investigate the effects of highlighting different job characteristics on the composition of the applicant pool. The survey experiment is conducted in cooperation with laboratories for economic research of universities across Germany.

Intervention

RCT within the tech firm:

In the RCT, we exogenously vary text blocks at the top of job ads.

These text blocks are part of a so called ‘teaser text’. The ‘teaser text’ is located at the top of the job ads and provides a high-level and general description of the advertised job and the firm itself.

We use a within-job-ad-randomization to identify treatment effects. In the control group, we keep the ‘teaser text’ as it is. In the first treatment (‘flexibility treatment’) we add a text block highlighting the opportunity for flexibility in the advertised position. The second treatment (‘growth treatment’) adds an additional text block (to the control treatment job ad) highlighting the opportunity for future wage growth.

Survey experiment :

The survey experiment is conducted in cooperation with several laboratories for economic research of Universities across Germany.

Experimental design

RCT within the tech firm:

In the RCT, we exogenously vary text blocks at the top of job ads.

These text blocks are part of a so called ‘teaser text’. The ‘teaser text’ is located at the top of the job ads and provides a high-level and general description of the advertised job and the firm itself.

We use a within-job-ad-randomization to identify treatment effects. In the control group, we keep the ‘teaser text’ as it is. In the first treatment (‘flexibility treatment’) we add a text block highlighting the opportunity for flexibility in the advertised position. The second treatment (‘growth treatment’) adds an additional text block (to the control treatment job ad) highlighting the opportunity for future wage growth.

Survey experiment:

In the survey, we show participants job ads which are part of the RCT described above. Thus, the job ads are ads for “real” jobs in a “real” company which invites applications for the position. The design of job ads shown to each respondent in the survey will vary (across randomization).

More specifically, we will randomly provide survey participants with one of three different job ads: the control treatment, the flexibility treatment, or the growth treatment.

In the survey we ask among others for demographic characteristics, educational background, preferences, ability beliefs and employment status. Moreover, we elicit participants’ expectations about the working conditions in the firm.

Specifically, we ask for the participants’ expectations regarding the working condition several times. The first time, we do not show them the location for which the job is advertised and ask them to assume that the job is located in a distance with a reasonable commuting time. Then we ask the expectation question again for in total two different cities in Germany (among which is also the true location for which the job is advertised). This allows us to check to which extent expectations are influenced via the location of the job.

Finally, we reveal the true location of the job and tell participants that the company invites applications for the position. Subsequently, we ask respondents whether they want to apply for the job and ask them to reason why or why they do not want to apply to the job. If they answer yes and they have an increased interest for the job, we offer them the opportunity to follow-up with an employee of the firm.

Design details

Randomization method
RCT within the tech firm: We use a within-randomization to identify treatment effects. The firm receives almost all the applications for jobs within the first 30 days of the job ad being posted online. In our study, we thus focus mainly on observations of the first 30 days. We randomly assign either the control treatment (no text block), the flexibility treatment or the growth treatment to the first ten days in which the job ad is posted online. Among the set of remaining treatments (excluding the one which was assigned for the ten days), we randomly assign one to the following ten-day period. The remaining treatment, which was not assigned to the first two ten-day blocks is assigned to the last ten-days. The job ad is online until the company agrees that they have received sufficiently many high-quality applications or the position is filled (which is usually within 30 days). If the job ad is online for more than 30 days, the treatment assigned to the last ten-days will be online until the job ad is put offline by the company. Survey experiment: We randomize across participants. More specifically, we will randomly provide survey participants with one of the three different job ads: the control treatment, the flexibility treatment, and the growth treatment.
Randomization unit
RCT within the tech firm: We randomize within job ads. Survey experiment: We randomize job ads across participants.
Clusters
n.a.
Observations
RCT within the tech firm: We start the RCT at the end of October 2022. The company confirmed that we can run the RCT until the end of October 2023; we aim to extent the RCT at least until the end of April 2024. We are not able to predict the exact number of observations in the RCT. Because of that, we summarize here the process how the data will be collected and our expected number of observations. Each year, the tech company is posting on average between 80 and 120 of job ads for engineering, business administration, IT and science (e.g., physics, biology) positions in the plant that cooperates with us. All of these job ads will be part of the RCT. For each job ad, the company receives approximately between 2 and 60 applications. In our RCT, one observation is one application. Standard errors will be clustered at the job ad level. We receive detailed data from the company about each application (e.g. gender, employment history, educational background) and data from the hiring process (e.g. classification of the quality of the applicant from the firm’s recruiters, which applicant is invited to an interview). Survey experiment: We seek to recruit survey participants whose educational background matches with the requirements listed in the job ad. In the laboratory pools, we select participants based on their field of study. All subjects whose field of study fits to the advertised position are invited to participate in the survey. Yet, we do not know how many suitable candidates are available in the laboratory pools, as we do not know which jobs will be advertised by the company during next year (and thus not the requirements). For 15-30 job ads (if possible) we will run the survey in the lab (the exact number depends on the total number of vacancies and the size of the subject pools / response rates in the labs). As there is uncertainty with regard to the required educational background needed for the positions and the positions are partly highly specialized, we do not know yet, whether we will be able to find sufficiently many suitable potential survey participants for the lab. Our aim is to collect for each job ad in total at least 45 observations (at least 15 survey participants for each treatment group), but we aim at 90 observations for each job ad (30 survey participants for each treatment group). Note that it is possible that this number will vary because of varying response rates in the different labs and because of the uncertainty with respect to the fit of the subjects’ background (job experience, education) to the job ads. We made some adjustments compared to our previous version of the preregistration (see the former version from November 18, 2022). Our initial plan was to run surveys for 20 job ads in cooperation with the market research agency Bilendi. However, as we started to pilot our surveys with them, we experienced a very low response quality. Simple quality checks (like reading the job ad for at least 20 seconds) led to a dropout of roughly 75% of the sample. Because of the low answer quality as well as potential selection bias resulting from it, we decided to stop working with Bilendi before analyzing the data; we conduct our whole survey via various econ labs across German speaking countries. Moreover, for statistical reasons, we decided to reduce the targeted number of job ads used in the survey for the sake of an increased number of participants in each survey wave. As the subject pool in the labs are mainly students with no or limited job experience, we are also planning to analyze in our RCT heterogeneous effects with respect to job experience.
Arms
n.a.
Primary outcomes
RCT within the tech firm: number of applications, number and share of female applicants, number and share of male applicants, number and share of non-native applicants Survey experiment: measures for applicants’ expectations regarding the working conditions, measures for the participants interest in the job
Secondary outcomes
RCT within the tech firm: fit of the application to the advertised position, quality of the applicant / application, commuting time to the current employer and our study firm’s plant, residence of the applicants, success of the applicants in the hiring process, probability that at least one woman applies, interaction of the various outcome variables, heterogeneous treatment effects with respect to the required work experience for the job Survey experiment: interaction of the various outcome variables

Cited by

No citation data available for this registration yet.

Keywords