An Experimental Study of AI Decision Support and Human Judgment

How to cite

Rønning, Kjetil Veen. 2026. "An Experimental Study of AI Decision Support and Human Judgment." AEA RCT Registry. AEARCTR-0019001. https://doi.org/10.1257/rct.19001-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Kjetil Veen Rønning*

* Primary investigator

Registry ID
AEARCTR-0019001
DOI
https://doi.org/10.1257/rct.19001-2.0
First registered
Last updated
Status
In development
Trial dates
2026-06-26 – 2026-11-02
Countries
Private
JEL codes
D91, C91, D83
Registry record
http://www.socialscienceregistry.org/trials/19001

Abstract

Artificial intelligence (AI) is increasingly deployed as decision support in organisations. This study examines how prior exposure to AI advice affects subsequent human judgment in case-based decision tasks.

The study is an online experiment with 1:1 individual randomisation. Participants complete three short decision cases involving managerial judgment under uncertainty. The target sample is at least 200 eligible participants with formal leadership responsibility, defined as at least one year of formal responsibility for leading, coordinating, or professionally directing the work of others.

In the treatment condition, participants complete all three cases in a sequential two-step format: they first provide an initial answer without AI support, then receive AI advice and are given the opportunity to revise their answer. In the control condition, participants complete Case 1 and Case 2 without AI support only. In Case 3, both groups follow the same two-step procedure by first providing an initial answer, then receiving AI advice and being given the opportunity to revise.

The primary outcome is an ordinal measure of whether, and how strongly, participants revise their Case 3 answer toward, or away from, the AI recommendation after seeing it. The primary test compares this revision between treatment and control using ordinal logistic regression on the full eligible sample, without conditioning on the initial answer. Secondary analyses examine decision quality in Cases 1 and 2, confidence change, perceived mental effort, perceived task difficulty, and time-on-task. All AI messages are pre-written and fixed across participants. No live language model is used.

Intervention

The intervention consists of pre-written AI decision-support messages embedded in an online decision-making experiment with participants who have formal leadership responsibility. Leadership responsibility is self-reported. Participants in the treatment condition receive AI advice after their initial answer in each of the three cases and may revise their answer. Participants in the control condition do not receive AI advice in Cases 1 and 2, but receive the same AI-advice procedure as treatment participants in Case 3. The purpose is to examine how prior exposure to AI decision support affects later reliance on AI advice.

Experimental design

Participants are recruited online through social media (e.g. LinkedIn), professional networks, and snowball distribution. Participants complete the experiment in a single online session. After consent and background questions, participants are randomised 1:1 to Treatment or Control at the start of the session.
All participants complete three short decision cases in a fixed order. The cases involve managerial judgment under uncertainty. In the treatment condition, participants provide an initial answer, report confidence, receive AI advice, and may revise their answer in each case. In the control condition, participants complete Cases 1 and 2 without AI support, and then follow the same two-step AI-advice procedure as treatment participants in Case 3.
The primary target sample is at least 200 eligible participants with formal leadership responsibility. Participants without leadership experience may complete the experiment but will be treated as a secondary convenience sample and are not included in the primary confirmatory analyses.
All AI messages are pre-written and fixed across participants within each condition. Participation is anonymous and voluntary; no personally identifying information is collected.

Design details

Randomization method
Randomisation is performed by the web-based experimental instrument at the start of each session. Allocation ratio is 1:1. Assignment is concealed from participants.
Randomization unit
Individual
Clusters
Not applicable
Observations
200 individuals with formal leadership responsibility, defined as at least one year of formal responsibility for leading, coordinating, or professionally directing the work of others. Recruitment will target this primary sample and data collection will continue until the pre-specified data collection end date. Participants without leadership experience may complete the experiment and may be retained as a secondary convenience sample, but they will not count toward the primary target sample and will not be included in the primary confirmatory analyses. No interim analysis of treatment effects will be used to determine whether recruitment continues or stops. Preregistration update 15.08.2026: Initial data collection end date was set to 15.08.2026, but was extended to 01.09.2026 due to slow recruitment (94 respondenter in target group at initial data collection end date).
Arms
Treatment arm: approximately 100 or more eligible leaders. Control arm: approximately 100 or more eligible leaders.
Primary outcomes
The primary outcome is an ordinal measure (move_c) of whether, and how strongly, participants revise their Case 3 answer toward, or away from, the AI recommendation after seeing it. The primary test compares this revision between treatment and control using ordinal logistic regression on the full eligible sample, with treatment arm as predictor. A pre-specified balance check on the initial-answer distribution gates an initial-answer-adjusted model, which is reported as a robustness analysis.
Secondary outcomes
• Performance in Cases 1 and 2: absolute deviation between the participant’s answer and a pre-specified normative benchmark in each case. For the treatment group, performance is based on the post-AI answer. For the control group, performance is based on the single unaided answer. • Within-treatment improvement in Cases 1 and 2: paired change in absolute deviation from pre-AI to post-AI answer. • Adoption of the Case 3 AI recommendation: a binary outcome coded 1 if the participant's post-advice answer is Option C and 0 otherwise, computed over the full eligible sample. • Confidence change: confidence ratings on a 1–7 scale collected before and after each case; confidence change is defined as post minus pre. • Subjective mental effort: mental effort rating on a 1–7 scale after each case. • Perceived task difficulty: perceived difficulty rating on a 1–7 scale after each case. • Time-on-task: time spent on each case. • Case 3 manipulation check: correct recall of the Case 3 AI recommendation. • AI advice reading time in Case 3.

Cited by

No citation data available for this registration yet.

Keywords