Testing the efficacy of AI tutoring in secondary mathematics: A scaled RCT in UK classrooms
How to cite
Brazão, Vasco. 2026. "Testing the efficacy of AI tutoring in secondary mathematics: A scaled RCT in UK classrooms." AEA RCT Registry. AEARCTR-0018079. https://doi.org/10.1257/rct.18079-1.1
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
One-to-one tutoring is widely regarded as the gold standard for personalised education, yet it remains prohibitively expensive to deliver at scale. Recent advances in generative AI have prompted growing interest in whether large language models (LLMs) can approximate the pedagogical effectiveness of expert human tutors. However, empirical evidence remains scarce. Most published evaluations of AI tutoring systems rely on user satisfaction metrics or short-duration laboratory tasks rather than rigorous field experimentation with validated learning outcome measures.
An exploratory randomised controlled trial (RCT) conducted in 2025 provided initial evidence that AI tutoring can support student mathematics learning at levels similar to expert human tutoring (N = 165; arxiv:2512.23633). In that trial, an AI tutor integrated into the Eedi mathematics platform and supervised by human tutors produced a 5.5 percentage-point increase in students’ likelihood of correctly answering novel questions relative to expert human tutoring alone. Supervising tutors approved 76% of the AI tutor’s drafted messages without substantive edits, and communicated general enthusiasm for the pedagogical quality and performance of the AI tutor.
The present trial scales and extends the design of the exploratory trial, with the aim of testing whether the initial findings generalise to a larger cohort, accumulate over a sustained intervention period, and translate to gains on an independent standardised assessment. We will recruit and randomize 1,200 students across UK secondary schools to four arms over a full school term. Students in the control arm will receive Eedi's standard remediation (static content and practice). Students in two AI tutoring arms will receive interactive tutoring from an AI system operating under expert supervision, with the arms differing in the contextual information provided to the AI: one will receive only session-level pedagogical context, while the other will receive longitudinal student information incorporating historical performance, prior misconceptions, and curriculum position. Students in the fourth arm will receive support from human expert tutors. Our primary outcome will be change on STAR Maths (Renaissance), a computer-adaptive standardised assessment administered at baseline and endline. Secondary outcomes will include: near-term measures of learning gains; self-report measures of student self-efficacy, motivation, and attitudes toward learning from errors; and tutor affect and experience across conditions.
Experimental design
We will recruit approximately 1,200 students in Years 8, 9, and 10 from 10 UK secondary schools. We will individually randomise students across 4 arms, with each student remaining in their single assigned arm for the full duration of the trial. The arms differ by what happens when a student submits a wrong answer to a check-in question while working on a quiz. Students in arm 1 (control) will receive the standard Eedi intervention with static (non-interactive) content, aimed at resolving the students' misconception and providing practice opportunities. Students in arms 2, 3, and 4 will instead start a tutoring conversation, also aimed at resolving the students' misconception and providing practice opportunities. In arms 2 and 3, the students will interact with an LLM whose generated messages will be supervised by expert human tutors. In arm 2, the LLM will receive session-level pedagogical context (the current question, the student's answer, and the identified misconception); in arm 3, the LLM will additionally receive longitudinal information about the student, including historical performance, prior misconceptions, and curriculum position. In arm 4, students will interact directly with an expert human tutor working on their own (without LLM support).
Tutors will be allocated to different arms on a daily basis, so that each tutor will experience all arms over the course of the trial.
Design details
Cited by
No citation data available for this registration yet.