Belief Updating under AI Advice: An Experiment
How to cite
XUE, Lian; Khanna, Manshu; Wei, Lijia; Wang, Ziyi. 2025. "Belief Updating under AI Advice: An Experiment." AEA RCT Registry. AEARCTR-0015153. https://doi.org/10.1257/rct.15153-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
The rapid integration of AI into healthcare presents a critical challenge: understanding how medical professionals incorporate AI advice into their diagnostic decision-making. While AI systems demonstrate increasing accuracy in medical diagnosis, their effectiveness ultimately depends on how humans update their beliefs in response to AI recommendations. We conduct a lab-in-the-field experiment examining belief updating processes when participants receive AI-generated medical advice. Our experimental design employs a quadratic scoring rule to elicit truthful reporting of beliefs across multiple treatment scenarios. Participants first provide prior probability estimates for treatment success, then report their expectations about the informativeness of upcoming AI advice, and finally update their beliefs after receiving AI recommendations. This three-stage elicitation, grounded in Bayesian updating frameworks, allows us to examine both the direct impact of AI advice on belief formation and the role of ex-ante trust in AI systems.
Intervention
This study examines how individuals update their beliefs when receiving AI-generated advice in medical decision-making scenarios. Participants evaluate treatment options across multiple hypothetical medical cases. For each case, participants: 1) provide their initial assessment of treatment success probabilities, 2) indicate their expectations about upcoming AI advice, and 3) make final assessments after receiving AI recommendations. The AI system used is ChatGPT-4o with a documented accuracy rate above 70%. All probability assessments are incentivized using proper scoring rules to encourage truthful reporting.
Experimental design
The experiment employs a 2×2 between-subjects factorial design investigating how individuals incorporate AI advice into medical decision-making. Participants evaluate treatment options across 15 hypothetical medical scenarios, providing probability estimates before and after receiving AI recommendations. The design uses incentive-compatible mechanisms to elicit truthful belief reporting. Treatments vary in the format of AI advice provided. All participants complete pre-experiment cognitive assessments and post-experiment questionnaires measuring algorithm literacy and trust.
Design details
Cited by
No citation data available for this registration yet.