Expert vs. AI Feedback: Evidence from a Large-Scale Math Experiment (2)

How to cite

Aulagnon, Raphaelle; Cristia, Julian; Cueto, Santiago; Malamud, Ofer. 2026. "Expert vs. AI Feedback: Evidence from a Large-Scale Math Experiment (2)." AEA RCT Registry. AEARCTR-0018782. https://doi.org/10.1257/rct.18782-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Raphaelle Aulagnon* , Julian Cristia , Santiago Cueto , Ofer Malamud

* Primary investigator

Registry ID
AEARCTR-0018782
DOI
https://doi.org/10.1257/rct.18782-1.0
First registered
Last updated
Status
In development
Trial dates
2026-06-01 – 2026-06-07
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/18782

Abstract

We evaluate how different forms of feedback delivered through a large-scale digital math platform affect the learning outcomes of primary school students in public schools in Peru. The platform “Apprendemos” currently provides weekly math activities consisting of 30 exercises to students across Peru in grades 3–6. In a one-week experiment, students will be individually randomized into one of three feedback conditions when completing the 30-item exercises: (i) status quo feedback indicating only whether an answer is correct or incorrect; (ii) expert-based feedback which provides explanations designed by a qualified team and that vary depending on the choice selected by the student; or (iii) AI-generated feedback, which provides explanations produced using a generative AI chatbot. For all students, items 1–5 will serve as a pre-test and items 26–30 as a post-test, while items 6–25 will be used to deliver the differentiated feedback.

Intervention

The intervention modifies the feedback that students receive while completing a 30-exercise math activity on the Apprendemos platform.
Control – Status quo Apprendemos feedback
For all 30 items, students receive the standard platform feedback, which simply indicates whether an answer is correct or incorrect.
Treatment 1 – Expert-based feedback
Items 1–5: standard Apprendemos feedback.
Items 6–25: rule-based feedback. Messages depend on the answers chosen for that particular item and are pre-specified by content experts.
Items 26–30: standard Apprendemos feedback.
Treatment 2 – AI feedback
Items 1–5: standard Apprendemos feedback.
Items 6–25: AI-generated feedback based on the student’s current response and past responses, produced via a generative AI model integrated into the platform.
Items 26–30: standard Apprendemos feedback.

Within the two treatment arms, feedback messages have the same structure:
If the student answers correctly on the first or second attempt, they receive the same “correct” message as in the status quo Apprendemos feedback (kept constant across arms).
On the first incorrect attempt, the student receives a hint. On the second incorrect attempt, the student receives an explanation and the correct answer.
Messages combine pedagogical content (e.g., explanations of the relevant math concept or procedure) and motivational content (e.g., encouragement to keep trying).
The intervention is implemented in collaboration with the Inter-American Development Bank (IDB), GRADE, and Apprendemos, with software development provided by the firm Pyxis to enable the three feedback modalities.

Experimental design

The experiment is embedded in the regular operation of the Apprendemos digital math platform for public primary schools in Peru.
The intervention consists of a single 30-exercise math activity delivered during one week.
Students are individually randomized into three groups of Control, Expert-Based and AI-Based Feedback.
The sample of users selected to participate in the experiment includes those who:
a. Do not take part in other special interventions in the Apprendemos program
b. Connected at least once in the first nine weeks of the year.
c. Use the app in Online mode (since this is necessary to view the generative AI feedback displayed).
d. Did not take part in the qualitative pilots.

Design details

Randomization method
Randomization was done using Stata on the sample selected (as mentioned above). a. Sample of 40,157 students meeting the requirements b. We randomly remove two students to have equally balanced group c. We divide the sample into three equal groups with no stratification design (13,385 each)
Randomization unit
Student
Clusters
40,155 students
Observations
40,155 students (40,155*30 questions)
Arms
13,385 control, 13,385 Expert Feedback, 13,385 AI Feedback
Primary outcomes
Academic achievement in the post test

Cited by

No citation data available for this registration yet.

Keywords