Human Experts and Artificial Intelligence: The Value of Human Input in Diagnostic Imaging (Teleradiology Version)

How to cite

Moehring, Alex; Agarwal, Nikhil; Salz, Tobias; Rajpurkar, Pranav. 2022. "Human Experts and Artificial Intelligence: The Value of Human Input in Diagnostic Imaging (Teleradiology Version)." AEA RCT Registry. AEARCTR-0009620. https://doi.org/10.1257/rct.9620-1.5

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Alex Moehring* , Nikhil Agarwal , Tobias Salz , Pranav Rajpurkar

* Primary investigator

Registry ID
AEARCTR-0009620
DOI
https://doi.org/10.1257/rct.9620-1.5
First registered
Last updated
Status
In development
Trial dates
2022-06-22 – 2023-12-31
Countries
United States of America ()
Registry record
http://www.socialscienceregistry.org/trials/9620

Abstract

We plan to investigate how human experts combine their own information with AI predictions when making assessments and decisions in the radiology domain. See the attached pre-analysis plan for full details. This study is a follow-on to the experiment described in AEA RCT Registry AEARCTR-0008799.

Intervention

See the attached pre-analysis plan for full details.

Experimental design

See the attached pre-analysis plan for full details of the experiment and analysis plan.

Design details

Randomization method
The attached pre-analysis plan contains the full randomization details. Radiologists will read patient cases in varying information environments. The following treatment arms will be cross-randomized. • AI Treatment: Patient cases are presented alongside assistance of an AI support tool or without it • Clinical history: Patient cases are presented alongside the patient's clinical history or without it • Incentive randomization: Responses will be incentivized or not
Randomization unit
Randomization occurs at the patient case level for each radiologist.
Clusters
We will aim to recruit 250 radiologists split into two groups, as described in the attached pre-analysis plan.
Observations
The first group of 200 radiologists will read 60 cases resulting in 12,000 total reads. The second group of 50 radiologists will read 100 cases total, resulting in 5,000 total reads.
Arms
The randomization will be balanced, so we expect half of the observations to be read with AI assistance, half to be read with clinical history, and half of responses to be incentivized. Note these are cross-randomized, so cases will be read with all combinations of AI assistance, clinical history, and incentives.
Primary outcomes
To measure the quality of diagnostic assessments and decisions we will focus on the following primary outcomes variables for each pathology group. 1. Error in probability assessment 2. Incorrect treatment/followup recommendation The primary pathology groups we will consider are: 1. Pooled outcomes for all pathologies 2. Pooled outcomes for all AI assisted pathologies 3. Pooled outcomes for all top-level AI assisted pathologies
Secondary outcomes
1. Time-taken and measures of effort exerted to parse the information in the X-ray and the clinical history, with and without AI 2. Treatment effects on distance from AI signal 3. Heterogeneity of treatment effects by pathology prevalence and AI performance

Cited by

No citation data available for this registration yet.

Keywords