The Effects of AI Feedback: Evidence from a Large-Scale Math Experiment

How to cite

Aulagnon, Raphaelle; Cristia, Julian; Malamud, Ofer; Cueto, Santiago. 2025. "The Effects of AI Feedback: Evidence from a Large-Scale Math Experiment." AEA RCT Registry. AEARCTR-0017399. https://doi.org/10.1257/rct.17399-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Raphaelle Aulagnon* , Julian Cristia , Ofer Malamud , Santiago Cueto

* Primary investigator

Registry ID
AEARCTR-0017399
DOI
https://doi.org/10.1257/rct.17399-1.0
First registered
Last updated
Status
In development
Trial dates
2025-12-08 – 2025-12-14
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/17399

Abstract

We evaluate how different forms of feedback delivered through a large-scale digital math platform affect the learning outcomes of primary school students in public schools in Peru. The platform “Apprendemos” currently provides weekly math activities consisting of 30 exercises to students across Peru in grades 3–6. In a one-week experiment, students will be individually randomized into one of two feedback conditions when completing the 30-item exercises: (i) status quo feedback indicating only whether an answer is correct or incorrect; (ii) AI-generated feedback, which provides explanations produced using a generative AI chatbot. For all students, items 1–5 will serve as a pre-test and items 26–30 as a post-test, while items 6–25 will be used to deliver the differentiated feedback.

Intervention

The intervention modifies the feedback that students receive while completing a 30-exercise math activity on the Apprendemos platform.
Control – Status quo Apprendemos feedback
For all 30 items, students receive the standard platform feedback, which simply indicates whether an answer is correct or incorrect.
Treatment – AI feedback
Items 1–5: standard Apprendemos feedback.
Items 6–25: AI-generated feedback based on the student’s current response and past responses, produced via a generative AI model integrated into the platform.
Items 26–30: standard Apprendemos feedback.
If the student answers correctly on the first or second attempt, they receive the same “correct” message as in the status quo Apprendemos feedback (kept constant across arms).
On the first incorrect attempt, the student receives a hint. On the second incorrect attempt, the student receives an explanation and the correct answer.
Messages combine pedagogical content (e.g., explanations of the relevant math concept or procedure) and motivational content (e.g., encouragement to keep trying).
The intervention is implemented in collaboration with the Inter-American Development Bank (IDB), GRADE, and Apprendemos, with software development provided by the firm Pyxis to enable the AI feedback treatment.

Experimental design

The experiment is embedded in the regular operation of the Apprendemos digital math platform for public primary schools in Peru.
The intervention consists of a single 30-exercise math activity delivered during one week.
Students are individually randomized into two groups of Control and AI-Based Feedback.
The sample of users selected to participate in the experiment includes those that:
a. Do not take part in other special interventions in the APPrendemos program
b. Connected at least once in the preceding year
c. Use the app in Online mode (since this is necessary to view the generative AI feedback displayed).

Design details

Randomization method
Randomization was done using Stata on the sample selected (as mentioned above). a. Sample of 43,003 students meeting the requirements b. We randomly remove one student to have equally balanced group c. We divide the sample into two equal groups with no stratification design
Randomization unit
Student
Clusters
43,002 students
Observations
43,002 students
Arms
21,501 control, 21,501 AI
Primary outcomes
Academic achievement in the post test. The main outcome is academic achievement measured using the post-test, defined as the number of correct answers in items 26–30. We will construct a standardized outcome by transforming the raw post-test score into a z-score with mean 0 and standard deviation 1 in the control group. All analyses will control for baseline performance, measured as the number of correct answers in the pre-test (items 1–5), also standardized using the distribution in the control group.

Cited by

No citation data available for this registration yet.

Keywords