An Online Experiment on Compliance to Recommendation Algorithms in a Hiring Task

How to cite

McLaughlin, Bryce; Spiess, Jann. 2023. "An Online Experiment on Compliance to Recommendation Algorithms in a Hiring Task." AEA RCT Registry. AEARCTR-0011857. https://doi.org/10.1257/rct.11857-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Bryce McLaughlin* , Jann Spiess

* Primary investigator

Registry ID
AEARCTR-0011857
DOI
https://doi.org/10.1257/rct.11857-1.0
First registered
Last updated
Status
In development
Trial dates
2023-08-10 – 2023-08-17
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/11857

Abstract

Our study uses an online lab experiment to understand how human decision-makers use recommendations provided by an algorithm, and how their use of the recommendation changes as the design of the algorithm varies. The experiment will assess if human participants can effectively use a recommendation algorithm which (i) ignores all observations for which the human has perfect certainty and/or (ii) withholds recommendations selectively. In theory these targeted recommendations are more effective than a generic recommendation as the participant should ignore the recommendation if they have perfect certainty (or aren't given the recommendation).

We have designed a small hiring game that will see if participants are able to properly ignore the algorithm when given certain information while taking the algorithm's recommendation in other instances. Participants see the role applicants are applying for (Engineering, Sales, or Communications) while a recommendation algorithm assisting them is able to assess their personality type. Participants need to hire good applicants and not hire bad applicants. Interventions vary the structure of the recommendation algorithm assisting the participants.

Intervention

Our Intervention is in the form of different recommendation algorithms given to the participant. We list the names of the treatments here for reference with the sample size distribution, but describe these treatments in the non-public section of the PAP.

Baseline
Compliance
Prior
Compliance-Prior
No Recommendation
Adjusted 3 Level
3 Level

Experimental design

Participants will be informed they are being placed in the role of a hiring manager who must make 25 assisted hiring decisions. Each candidate (all of which are hypothetical) is good or bad and the good candidates should be hired while the bad candidates should not be hired. The participant knows the role the individual is interviewing for (Engineering, Sales, or Communications) and the probability the candidate is good for that role based on the pool your company hires from (100%, 60%, 40%). Participants are informed that to earn the most money they should hire if and only if they believe the probability a candidate is good is 50% or higher.

Participant answer comprehension questions.

In addition the participant may receive access to a recommendation algorithm (differentiated by treatments) that recommends they hire or don't hire the candidate. This algorithm sees a different attribute of the candidate than the role they are hiring for and then makes a report of (according to the algorithms information and the signaling mechanism described by the participants condition) hire, not hire, or no recommendation made.

Participant answer comprehension questions.

Participants then make 25 hiring decisions with their recommendation algorithm. For each decision they see what the recommendation says and what role they are hiring for. They do not see the personality type. To generate these candidates we will take a 25 member applicant pool (given to the participants) and give these 25 candidates to the participant in a random order.

Design details

Randomization method
Randomization will be done through the qualtrics randomizer which will balance (in weights we choose) the treatment distribution to incoming participants
Randomization unit
Randomization is performed at the study subject level.
Clusters
1500 subject
Observations
1500 subjects each making 25 decisions.
Arms
Baseline - 250 participants Compliance - 250 participants Prior - 250 participants Compliance Prior - 250 participants No Recommendation - 125 participants Adjusted 3 Level - 250 participants 3 Level - 125 participants
Primary outcomes
The difference in performance on our hiring task between participants who received different recommendation algorithms. Performance for us is defined as (Number of Good applicants Hired) + (Number of Bad applicants Not Hired). Specifically our hypotheses are: H1 - The average performance of the Compliance group exceeds the average performance of the Baseline group. H2 - The average performance of the Prior group exceeds the average performance of the Baseline group. H3a- The average performance of the Compliance-Prior group exceeds the average performance of the Prior group. H3b- The average performance of the Compliance-Prior group exceeds the average performance of the Compliance group. H4 - The average performance of the Compliance-Prior group exceeds the average performance of the Adjusted 3 Level group.
Secondary outcomes
The probability each candidate is hired in each intervention group.

Cited by

No citation data available for this registration yet.

Keywords