The Effects of AI Feedback: Evidence from a Large-Scale Math Experiment
How to cite
Aulagnon, Raphaelle; Cristia, Julian; Malamud, Ofer; Cueto, Santiago. 2025. "The Effects of AI Feedback: Evidence from a Large-Scale Math Experiment." AEA RCT Registry. AEARCTR-0017399. https://doi.org/10.1257/rct.17399-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We evaluate how different forms of feedback delivered through a large-scale digital math platform affect the learning outcomes of primary school students in public schools in Peru. The platform “Apprendemos” currently provides weekly math activities consisting of 30 exercises to students across Peru in grades 3–6. In a one-week experiment, students will be individually randomized into one of two feedback conditions when completing the 30-item exercises: (i) status quo feedback indicating only whether an answer is correct or incorrect; (ii) AI-generated feedback, which provides explanations produced using a generative AI chatbot. For all students, items 1–5 will serve as a pre-test and items 26–30 as a post-test, while items 6–25 will be used to deliver the differentiated feedback.
Intervention
The intervention modifies the feedback that students receive while completing a 30-exercise math activity on the Apprendemos platform.
Control – Status quo Apprendemos feedback
For all 30 items, students receive the standard platform feedback, which simply indicates whether an answer is correct or incorrect.
Treatment – AI feedback
Items 1–5: standard Apprendemos feedback.
Items 6–25: AI-generated feedback based on the student’s current response and past responses, produced via a generative AI model integrated into the platform.
Items 26–30: standard Apprendemos feedback.
If the student answers correctly on the first or second attempt, they receive the same “correct” message as in the status quo Apprendemos feedback (kept constant across arms).
On the first incorrect attempt, the student receives a hint. On the second incorrect attempt, the student receives an explanation and the correct answer.
Messages combine pedagogical content (e.g., explanations of the relevant math concept or procedure) and motivational content (e.g., encouragement to keep trying).
The intervention is implemented in collaboration with the Inter-American Development Bank (IDB), GRADE, and Apprendemos, with software development provided by the firm Pyxis to enable the AI feedback treatment.
Experimental design
The experiment is embedded in the regular operation of the Apprendemos digital math platform for public primary schools in Peru.
The intervention consists of a single 30-exercise math activity delivered during one week.
Students are individually randomized into two groups of Control and AI-Based Feedback.
The sample of users selected to participate in the experiment includes those that:
a. Do not take part in other special interventions in the APPrendemos program
b. Connected at least once in the preceding year
c. Use the app in Online mode (since this is necessary to view the generative AI feedback displayed).
Design details
Cited by
No citation data available for this registration yet.