NUMI: A Within-Class Randomized Evaluation of AI-Tutoring in Mastery-Based Computer-Assisted Math Learning
How to cite
Oreopoulos, Philip; Liut, Michael. 2026. "NUMI: A Within-Class Randomized Evaluation of AI-Tutoring in Mastery-Based Computer-Assisted Math Learning." AEA RCT Registry. AEARCTR-0018643. https://doi.org/10.1257/rct.18643-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
There is growing interest in whether Generative AI has the potential to transform education and offer personalized learning at scale, but early evidence suggests students often fail to engage productively with AI tutors. This study tests the causal value-added of AI tutoring over a well-implemented Computer-Assisted Learning (CAL) platform, and identifies which AI-design features increase productive engagement. We deploy NUMI, a research-purpose CAL platform with a guard-railed AI math tutor, in Grades 4–9 classrooms in the Hamilton County Department of Education (HCDE) over the 2026–27 school year. All students use NUMI in its Mastery mode (must answer three problems in a row correctly to advance) on a weekly exercise. Within each participating classroom, students are randomized to one of two sequences: AI in Trimester 1, no-AI in Trimester 2 (A-N-A); or no-AI in Trimester 1, AI in Trimester 2 (N-A-A). All students receive AI in Trimester 3 for fairness. The design supports a parallel-arm contrast in Trimester 1 and a within-student crossover estimate pooling Trimesters 1 and 2, with explicit tests for carryover. Within the AI condition, embedded randomized comparisons test pre-specified tutor design variations (e.g., conversational tone, response-mode prompts). Outcomes include immediate performance (next-question correctness, exit tickets), near-term retention (improvement checks at 2–4 week lag), and administrative outcomes (course grades, NWEA MAP, state assessments). Heterogeneity analyses focus on baseline achievement and disadvantaged subgroups.
Intervention
Each week, participating teachers in Grades 4–9 assign a NUMI module: one curriculum-aligned exercise with short instructional content and practice problems, ending in a brief exit ticket worth a participation grade. Unfinished work is completed at home or during extra time within the week. All students use NUMI in Mastery mode, meaning they must answer three problems in a row correctly before advancing within the exercise.
The intervention manipulates one feature: access to an AI tutor. In the no-AI condition, students who answer incorrectly are shown a step-by-step written solution. In the AI condition, students are offered access to NUMI, a domain-bounded AI math tutor that elicits the student's reasoning rather than revealing the final answer; walks through solution steps interactively using yes/no, multiple-choice, or short-answer prompts; offers a "Help Me Get Started" option for first-step scaffolding; and after a mistake suggests likely misconceptions and walks through the worked solution while prompting comprehension checks. The AI tutor operates within math-only content filters, has access to the canonical step-by-step solution as a grounding reference, and stores only non-identifiable session metadata.
Every 2–4 weeks, teachers administer a short in-class Improvement Check that revisits prior content to measure retention. The intervention runs across three trimesters of the 2026–27 school year.
Experimental design
Within-classroom, student-level two-period crossover design with universal phase-in, conducted across three trimesters of the 2026–27 school year.
All students use NUMI in Mastery mode on weekly math exercises. Within each participating classroom, students are randomly assigned to one of two sequences:
Sequence A-N-A: AI tutor in Trimester 1, no-AI (step-by-step solutions only) in Trimester 2, AI in Trimester 3.
Sequence N-A-A: no-AI in Trimester 1, AI in Trimester 2, AI in Trimester 3.
The Trimester 1 comparison provides a parallel-arm contrast between the two conditions. Pooling Trimesters 1 and 2 with student fixed effects provides a crossover estimate of the AI effect, with explicit testing for carryover. Trimester 3 is universal AI: it does not provide a contemporaneous treatment contrast, but ensures every student receives AI in two of three trimesters (a fairness consideration with the partnering district), and supports analyses of persistence and implementation under universal access. Within the AI condition, embedded randomized comparisons test pre-specified tutor design features.
Design details
Cited by
No citation data available for this registration yet.