Can Feedback from a Large Language Model Improve Health Care Quality?
How to cite
Sautmann, Anja; Abaluck, Jason; Schwartz, Aaron; Pless, Robert; Ravi, Nirmal. 2025. "Can Feedback from a Large Language Model Improve Health Care Quality?." AEA RCT Registry. AEARCTR-0015226. https://doi.org/10.1257/rct.15226-1.1
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
The quality of healthcare in low- and middle-income countries is notoriously low (Banerjee et al., 2023). In Nigeria, only a small proportion of primary care patients are seen by a Medical Officer (MO) instead of a mid-level provider such as a Community Health Extension Worker. Access to qualified medical doctors is a barrier to good health care (Okeke, 2023). This project tests whether Large Language Models (LLMs) can improve patient care in Nigerian primary care clinics. An LLM-based tool provides "second opinions'' to community health extension workers (CHEWs) at two clinics in Nigeria. These second opinions are intended to mirror what a reviewing physician might advise the CHEWs after seeing or hearing their initial report on a patient. For our main analysis, we use a within-patient comparison of two patient notes created by the CHEW; one during the initial patient consultation, and one after the LLM feedback was received. The patient is also seen by a fully trained MO who is in charge of patient care. The MO conducts a blinded review of the CHEW's patient notes to measures changes in the CHEW's care as a result of the LLM feedback.
Intervention
An LLM is prompt-engineered to provide specific and concrete feedback on structured patient notes created by community health extension workers (CHEWs) at two clinics in Nigeria as part of their usual data entry into an electronic patient management system.
Experimental design
This project tests whether Large Language Models (LLMs) can improve patient care in Nigerian primary
care clinics by giving customized and instant feedback to the provider in natural language. An LLM-based
tool integrated into an electronic patient management system provides “second opinions” to community
health extension workers (CHEWs) at two clinics in Nigeria. These second opinions are intended to mirror
what a reviewing physician might advise the CHEWs after seeing or hearing their initial report on a patient.
For our main analysis, we use a within-patient comparison of two patient notes created by the CHEW; one
during the initial patient consultation, and one after the LLM feedback was received. The patient is also
seen by a fully trained medical officer who is in charge of patient care. The MO conducts a randomized blinded review
of the CHEW’s patient notes to measures changes in the CHEW’s care as a result of the LLM feedback.
Design details
Materials
Papers from this trial
- Jason Abaluck, Robert Pless, Nirmal Ravi, Anja Sautmann, and Aaron Schwartz, "Does LLM Assistance Improve Healthcare Delivery? An Evaluation Using On-site Physicians and Laboratory Tests," NBER Working Paper 34660 (2026), https://doi.org/10.3386/w34660. https://doi.org/10.3386/w34660
Cited by
No citation data available for this registration yet.