Online Survey to Test Whether Different Prompts Elicit Different Responses to Potential Job Applicants

How to cite

Agan, Amanda; Cowgill, Bo; Gee, Laura. 2024. "Online Survey to Test Whether Different Prompts Elicit Different Responses to Potential Job Applicants." AEA RCT Registry. AEARCTR-0014133. https://doi.org/10.1257/rct.14133-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Amanda Agan* , Bo Cowgill , Laura Gee

* Primary investigator

Registry ID
AEARCTR-0014133
DOI
https://doi.org/10.1257/rct.14133-1.1
First registered
Last updated
Status
In development
Trial dates
2024-08-12 – 2025-08-11
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/14133

Abstract

In surveys, social desirability bias can impact how respondents answer questions. We intend to survey people to ask their responses to hypothetical scenarios where we randomize language about the scenario in which they are answering the question. We will use their responses to understand how different language and prompts about the situation in which they are responding to the questions may (or may not) change how they people answer questions.

Intervention

Respondents will be answering a hypothetical survey with vignette style hiring scenarios. There are several interventions:

Anonymity intervention: Some respondents will be randomized to receive a treatment that reminds them about their anonymity and the confidentiality of their responses. The other half will not receive this message.

One-time job intervention: Some respondents, when answering the vignette survey, will be told that the company they are hypothetically hiring for has stated this is a one-time job, while the other half will be told that they company may possibly hire them in the future for a similar task.

Academic study intervention: Some respondents, when answering the vignette survey, will be told the hypothetical company would review them based on their performance as a subject in an academic study. The others will be told their review would be based on performance as a hiring professional (even though they are participating in an academic study).

Generally the set of candidates that the respondents will review will also be randomized, to understand the effect this may have on results, but it is not considered by the researchers as an 'intervention'

Experimental design

For each intervention, survey respondents will be (independently) randomly assigned to receive one of the treatments listed in the intervention explanation above.

Design details

Randomization method
Randomization done by Qualtrics
Randomization unit
Individual
Clusters
No clusters
Observations
500
Arms
250 receiving each intervention
Primary outcomes
Each respondent is being asked to choose between candidate A or candidate B in several vignettes. Candidate B is usually has typically non-discriminated against characteristics (e.g. white male with no criminal record) while Candidate A may be female, Black, or have a criminal record. For the anonymity intervention 1, one primary outcome is whether they choose candidate A or B. For the anonymity intervention 2, one-time job and academic study interventions, the primary outcomes are: how comfortable would you feel choosing candidate B (on a sliding scale from very uncomfortable to very comfortable) and how much effort would you put into choosing between candidate A and B (on a sliding scale from very little effort to a lot of effort)
Secondary outcomes
Time spent on making a decision (measured by how much time Qualtrics says they stayed on that page)

Cited by

No citation data available for this registration yet.

Keywords