Can Feedback from a Large Language Model Improve Health Care Quality?

How to cite

Sautmann, Anja; Abaluck, Jason; Schwartz, Aaron; Pless, Robert; Ravi, Nirmal. 2025. "Can Feedback from a Large Language Model Improve Health Care Quality?." AEA RCT Registry. AEARCTR-0015226. https://doi.org/10.1257/rct.15226-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Anja Sautmann* , Jason Abaluck , Aaron Schwartz , Robert Pless , Nirmal Ravi

* Primary investigator

Registry ID
AEARCTR-0015226
DOI
https://doi.org/10.1257/rct.15226-1.1
First registered
Last updated
Status
Completed
Trial dates
2025-01-30 – 2025-10-17
Countries
Nigeria (Kano)
JEL codes
I10, O12, O15
Registry record
http://www.socialscienceregistry.org/trials/15226

Abstract

The quality of healthcare in low- and middle-income countries is notoriously low (Banerjee et al., 2023). In Nigeria, only a small proportion of primary care patients are seen by a Medical Officer (MO) instead of a mid-level provider such as a Community Health Extension Worker. Access to qualified medical doctors is a barrier to good health care (Okeke, 2023). This project tests whether Large Language Models (LLMs) can improve patient care in Nigerian primary care clinics. An LLM-based tool provides "second opinions'' to community health extension workers (CHEWs) at two clinics in Nigeria. These second opinions are intended to mirror what a reviewing physician might advise the CHEWs after seeing or hearing their initial report on a patient. For our main analysis, we use a within-patient comparison of two patient notes created by the CHEW; one during the initial patient consultation, and one after the LLM feedback was received. The patient is also seen by a fully trained MO who is in charge of patient care. The MO conducts a blinded review of the CHEW's patient notes to measures changes in the CHEW's care as a result of the LLM feedback.

Intervention

An LLM is prompt-engineered to provide specific and concrete feedback on structured patient notes created by community health extension workers (CHEWs) at two clinics in Nigeria as part of their usual data entry into an electronic patient management system.

Experimental design

This project tests whether Large Language Models (LLMs) can improve patient care in Nigerian primary
care clinics by giving customized and instant feedback to the provider in natural language. An LLM-based
tool integrated into an electronic patient management system provides “second opinions” to community
health extension workers (CHEWs) at two clinics in Nigeria. These second opinions are intended to mirror
what a reviewing physician might advise the CHEWs after seeing or hearing their initial report on a patient.
For our main analysis, we use a within-patient comparison of two patient notes created by the CHEW; one
during the initial patient consultation, and one after the LLM feedback was received. The patient is also
seen by a fully trained medical officer who is in charge of patient care. The MO conducts a randomized blinded review
of the CHEW’s patient notes to measures changes in the CHEW’s care as a result of the LLM feedback.

Design details

Randomization method
simple randomization integrated into data interface
Randomization unit
Patient SOAP note (provider's patient encounter record)
Clusters
500 patients with 2 SOAP notes each.
Observations
1000 (2 per patient)
Arms
500 per arm (2 per patient, unassisted and LLM-assisted patient note)
Primary outcomes
Indicator for "any treatment error", indicator for "severe treatment error", indicator for "better treatment plan", and a treatment misallocation indicator for malaria, anemia, and UTI (see pre-analysis plan for details)
Secondary outcomes
See pre-analysis plan.
Sponsors
Name: World Bank Location: Washington, DC Url: https://www.worldbank.org/ Name: Bill and Melinda Gates Foundation Url: https://gcgh.grandchallenges.org/grant/large-language-model-llm-tool-support-frontline-health-workers-low-resource-settings
Partners
Name: EHA Clinics Partner_type: private_company Url: https://www.eha.ng

Materials

Papers from this trial

  1. Jason Abaluck, Robert Pless, Nirmal Ravi, Anja Sautmann, and Aaron Schwartz, "Does LLM Assistance Improve Healthcare Delivery? An Evaluation Using On-site Physicians and Laboratory Tests," NBER Working Paper 34660 (2026), https://doi.org/10.3386/w34660. https://doi.org/10.3386/w34660

Cited by

No citation data available for this registration yet.

Keywords