Diagnostic Reasoning With and Without AI Support: A Randomized Controlled Trial of LLM-Trained Medical Doctors

How to cite

Qazi, Ihsan Ayyub; Ali, Ayesha; Khawaja, Muhammad Asadullah; Sheikh, Ali Zafar; Akhtar, Muhammad Junaid; Alizai, Muhammad Hamad. 2025. "Diagnostic Reasoning With and Without AI Support: A Randomized Controlled Trial of LLM-Trained Medical Doctors." AEA RCT Registry. AEARCTR-0015117. https://doi.org/10.1257/rct.15117-1.2

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Ihsan Ayyub Qazi* , Ayesha Ali , Muhammad Asadullah Khawaja , Ali Zafar Sheikh , Muhammad Junaid Akhtar , Muhammad Hamad Alizai

* Primary investigator

Registry ID
AEARCTR-0015117
DOI
https://doi.org/10.1257/rct.15117-1.2
First registered
Last updated
Status
In development
Trial dates
2025-01-10 – 2025-09-30
Countries
Private
JEL codes
I11, C91, J24, D83
Registry record
http://www.socialscienceregistry.org/trials/15117

Abstract

Diagnostic errors are a major source of preventable patient harm. Large language models (LLMs) have shown promise in assisting with clinical decision-making, potentially improving diagnostic accuracy and efficiency. However, the impact of LLMs on medical doctors' diagnostic reasoning compared to conventional diagnostic resources remains unclear. This study aims to evaluate whether providing LLM-trained medical doctors (including physicians and surgeons) with access to ChatGPT-4o, in addition to standard resources, enhances their diagnostic reasoning performance. All participating doctors will have completed at least a 10-hour training program covering ChatGPT-4o usage, prompt engineering techniques, and output evaluation strategies.

Intervention

Treatment group will be given access to ChatGPT-4o.

Experimental design

The trial will be designed as a randomized, two-arm, single-blind parallel group study.

Design details

Randomization method
Randomization done in office by a computer
Randomization unit
Individual medical doctor.
Clusters
No clustering. Individual medical doctors will be randomized.
Observations
50 medical doctors
Arms
25 medical doctors each in both treatment and control
Primary outcomes
The primary outcome will be the diagnostic reasoning score, calculated as a percentage score (0-100%) for each case.
Secondary outcomes
Time Spent on Diagnosis

Cited by

No citation data available for this registration yet.

Keywords