Testing the efficacy of AI tutoring in secondary mathematics: A scaled RCT in UK classrooms

How to cite

Brazão, Vasco. 2026. "Testing the efficacy of AI tutoring in secondary mathematics: A scaled RCT in UK classrooms." AEA RCT Registry. AEARCTR-0018079. https://doi.org/10.1257/rct.18079-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Vasco Brazão*

* Primary investigator

Registry ID
AEARCTR-0018079
DOI
https://doi.org/10.1257/rct.18079-1.1
First registered
Last updated
Status
In development
Trial dates
2026-03-18 – 2026-08-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/18079

Abstract

One-to-one tutoring is widely regarded as the gold standard for personalised education, yet it remains prohibitively expensive to deliver at scale. Recent advances in generative AI have prompted growing interest in whether large language models (LLMs) can approximate the pedagogical effectiveness of expert human tutors. However, empirical evidence remains scarce. Most published evaluations of AI tutoring systems rely on user satisfaction metrics or short-duration laboratory tasks rather than rigorous field experimentation with validated learning outcome measures.

An exploratory randomised controlled trial (RCT) conducted in 2025 provided initial evidence that AI tutoring can support student mathematics learning at levels similar to expert human tutoring (N = 165; arxiv:2512.23633). In that trial, an AI tutor integrated into the Eedi mathematics platform and supervised by human tutors produced a 5.5 percentage-point increase in students’ likelihood of correctly answering novel questions relative to expert human tutoring alone. Supervising tutors approved 76% of the AI tutor’s drafted messages without substantive edits, and communicated general enthusiasm for the pedagogical quality and performance of the AI tutor.
The present trial scales and extends the design of the exploratory trial, with the aim of testing whether the initial findings generalise to a larger cohort, accumulate over a sustained intervention period, and translate to gains on an independent standardised assessment. We will recruit and randomize 1,200 students across UK secondary schools to four arms over a full school term. Students in the control arm will receive Eedi's standard remediation (static content and practice). Students in two AI tutoring arms will receive interactive tutoring from an AI system operating under expert supervision, with the arms differing in the contextual information provided to the AI: one will receive only session-level pedagogical context, while the other will receive longitudinal student information incorporating historical performance, prior misconceptions, and curriculum position. Students in the fourth arm will receive support from human expert tutors. Our primary outcome will be change on STAR Maths (Renaissance), a computer-adaptive standardised assessment administered at baseline and endline. Secondary outcomes will include: near-term measures of learning gains; self-report measures of student self-efficacy, motivation, and attitudes toward learning from errors; and tutor affect and experience across conditions.

Experimental design

We will recruit approximately 1,200 students in Years 8, 9, and 10 from 10 UK secondary schools. We will individually randomise students across 4 arms, with each student remaining in their single assigned arm for the full duration of the trial. The arms differ by what happens when a student submits a wrong answer to a check-in question while working on a quiz. Students in arm 1 (control) will receive the standard Eedi intervention with static (non-interactive) content, aimed at resolving the students' misconception and providing practice opportunities. Students in arms 2, 3, and 4 will instead start a tutoring conversation, also aimed at resolving the students' misconception and providing practice opportunities. In arms 2 and 3, the students will interact with an LLM whose generated messages will be supervised by expert human tutors. In arm 2, the LLM will receive session-level pedagogical context (the current question, the student's answer, and the identified misconception); in arm 3, the LLM will additionally receive longitudinal information about the student, including historical performance, prior misconceptions, and curriculum position. In arm 4, students will interact directly with an expert human tutor working on their own (without LLM support).

Tutors will be allocated to different arms on a daily basis, so that each tutor will experience all arms over the course of the trial.

Design details

Randomization method
Students will be machine-randomised whereby each Eedi UserId will be assigned to one condition by randomly permuting a vector of strings representing arms, where each arm appears the pre-specified number of times, using an R script and a seed for reproducibility.
Randomization unit
Individual
Clusters
1,200 students
Observations
1,200 students
Arms
Arm 1: 225; Arm 2: 375; Arm 3: 375; Arm 4: 225
Primary outcomes
Student scores on the STAR Renaissance Maths assessment, adjusted for baseline scores.
Secondary outcomes
Misconception resolution; near-term knowledge transfer; interim quiz success

Cited by

No citation data available for this registration yet.

Keywords