NUMI: An AI-Tutor for Learning How Best to Deliver Transformative Personalized Education at Scale
How to cite
Oreopoulos, Philip; Liut, Michael. 2026. "NUMI: An AI-Tutor for Learning How Best to Deliver Transformative Personalized Education at Scale." AEA RCT Registry. AEARCTR-0018678. https://doi.org/10.1257/rct.18678-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
There is growing interest in whether Generative AI technology has the potential to transform education and offer personalized learning for all. Early research, however, shows challenges with implementation and structure for getting students to use it effectively. To test AI's potential and develop best practices for engaging students, we propose a student-level, within-class randomized evaluation of NUMI, an online platform designed for research that pairs mastery pacing with a guard-railed AI math tutor. In partnership with the Hamilton County Department of Education (HCDE) and at least 50-70 Grades 4-9 teachers (~1,500-2,100 students), each trimester students are randomly assigned in a 2x2 design — Mastery vs. no Mastery; AI-Tutor vs. no AI-Tutor. We will measure immediate learning on lagged "improvement checks," behavioral mechanisms such as persistence after mistakes and time-on-task, and district outcomes including course grades and, where available, standardized assessments. Embedded randomized tests within the AI arms will allow us to assess a limited number of pre-specified design variations, such as prompting style and timing of support, in order to identify which forms of AI guidance increase productive engagement and learning. The study builds on previous J-PAL collaborations and pivots to a flexible, customizable platform that will causally identify the value-added of AI tutoring over well-implemented CAL, and inform policy makers and program designers which AI and structural features deliver the largest durable, equitable gains at scale.
Intervention
The intervention is NUMI, a research-oriented computer-assisted learning (CAL) platform that delivers weekly math practice to Grades 4-9 students. Each week, teachers assign a 30-90 minute NUMI assignment consisting of short instructional videos, exercises, and a brief exit-ticket assessment that contributes to a participation grade. Students who do not finish in class complete the remainder at home or during extra time.
Students are randomized at the individual level within their classroom to one of four conditions in a 2x2 factorial design:
- CAL only (control): non-Mastery progression, no AI tutor, step-by-step solutions on mistakes.
- CAL + AI: non-Mastery progression, plus access to NUMI's guard-railed AI tutor.
- CAL + Mastery: must answer 3 problems correctly in a row before advancing, no AI tutor.
- CAL + Mastery + AI: Mastery progression plus AI tutor.
In Mastery mode, students must answer three problems in a row correctly before advancing to the next exercise; in non-Mastery mode, students decide when they feel ready to take the exit ticket. In the AI arms, the AI tutor becomes available in a safe, domain-bounded chat space when a student makes a mistake or requests help. It elicits the student's reasoning and walks through steps together without revealing the final answer. NUMI primarily poses questions requiring binary (yes/no) or multiple-choice responses, with an occasional "Help Me Get Started" option (option A/B). After a student errs, NUMI suggests the likely misconception, reviews the worked solution step-by-step, and prompts for further questions. A text box is available for open-ended questions; filters and classifiers suppress off-topic or personal content. No student inputs are stored by any external AI provider. In the non-AI arms, students see step-by-step worked solutions when they make a mistake.
Trimester rotation: Each student rotates conditions across the three trimesters, experiencing three of the four conditions, never repeating. The rotation is pre-randomized at the start of the year.
Embedded A/B test within AI arms (pre-specified): Students in the two AI arms are independently randomized to either standard prompting or a humor / relatable-examples variant. Additional embedded A/B tests may be added in registry updates before each trimester begins.
Improvement checks: Every 2-4 weeks, teachers deliver short in-class "improvement check" assessments that revisit prior content after a lag to gauge retention.
Experimental design
A within-classroom, student-level randomized 2x2 factorial design (Mastery x AI-Tutor) with trimester-level rotation, conducted across Grades 4-9 math classes in the Hamilton County Department of Education (HCDE) in Chattanooga, Tennessee. Classrooms serve as the stratification block; students within a classroom are randomly assigned by the NUMI platform to one of the four arms (CAL-only; CAL+AI; CAL+Mastery; CAL+Mastery+AI), then rotate through three of four arms across the school year's three trimesters. A secondary, factorial randomization within the AI arms assigns students to humor / relatable-examples prompting vs. standard prompting.
Design details
Cited by
No citation data available for this registration yet.