An Experimental Study of AI Decision Support and Human Judgment
How to cite
Rønning, Kjetil Veen. 2026. "An Experimental Study of AI Decision Support and Human Judgment." AEA RCT Registry. AEARCTR-0019001. https://doi.org/10.1257/rct.19001-2.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Artificial intelligence (AI) is increasingly deployed as decision support in organisations. This study examines how prior exposure to AI advice affects subsequent human judgment in case-based decision tasks.
The study is an online experiment with 1:1 individual randomisation. Participants complete three short decision cases involving managerial judgment under uncertainty. The target sample is at least 200 eligible participants with formal leadership responsibility, defined as at least one year of formal responsibility for leading, coordinating, or professionally directing the work of others.
In the treatment condition, participants complete all three cases in a sequential two-step format: they first provide an initial answer without AI support, then receive AI advice and are given the opportunity to revise their answer. In the control condition, participants complete Case 1 and Case 2 without AI support only. In Case 3, both groups follow the same two-step procedure by first providing an initial answer, then receiving AI advice and being given the opportunity to revise.
The primary outcome is an ordinal measure of whether, and how strongly, participants revise their Case 3 answer toward, or away from, the AI recommendation after seeing it. The primary test compares this revision between treatment and control using ordinal logistic regression on the full eligible sample, without conditioning on the initial answer. Secondary analyses examine decision quality in Cases 1 and 2, confidence change, perceived mental effort, perceived task difficulty, and time-on-task. All AI messages are pre-written and fixed across participants. No live language model is used.
Intervention
The intervention consists of pre-written AI decision-support messages embedded in an online decision-making experiment with participants who have formal leadership responsibility. Leadership responsibility is self-reported. Participants in the treatment condition receive AI advice after their initial answer in each of the three cases and may revise their answer. Participants in the control condition do not receive AI advice in Cases 1 and 2, but receive the same AI-advice procedure as treatment participants in Case 3. The purpose is to examine how prior exposure to AI decision support affects later reliance on AI advice.
Experimental design
Participants are recruited online through social media (e.g. LinkedIn), professional networks, and snowball distribution. Participants complete the experiment in a single online session. After consent and background questions, participants are randomised 1:1 to Treatment or Control at the start of the session.
All participants complete three short decision cases in a fixed order. The cases involve managerial judgment under uncertainty. In the treatment condition, participants provide an initial answer, report confidence, receive AI advice, and may revise their answer in each case. In the control condition, participants complete Cases 1 and 2 without AI support, and then follow the same two-step AI-advice procedure as treatment participants in Case 3.
The primary target sample is at least 200 eligible participants with formal leadership responsibility. Participants without leadership experience may complete the experiment but will be treated as a secondary convenience sample and are not included in the primary confirmatory analyses.
All AI messages are pre-written and fixed across participants within each condition. Participation is anonymous and voluntary; no personally identifying information is collected.
Design details
Cited by
No citation data available for this registration yet.