Learning with Machines: A Randomized Evaluation of AI-Assisted Education in Schools

How to cite

Liu, Elaine; Zuo, Sharon Xuejing; Chen, Zhao; Li, Kaiyue; Zhong, Yuelin. 2026. "Learning with Machines: A Randomized Evaluation of AI-Assisted Education in Schools." AEA RCT Registry. AEARCTR-0018533. https://doi.org/10.1257/rct.18533-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Elaine Liu* , Sharon Xuejing Zuo , Zhao Chen , Kaiyue Li , Yuelin Zhong

* Primary investigator

Registry ID
AEARCTR-0018533
DOI
https://doi.org/10.1257/rct.18533-1.0
First registered
Last updated
Status
On going
Trial dates
2026-03-23 – 2026-10-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/18533

Abstract

This study evaluates the effectiveness of a technology-based learning intervention in rural middle schools using a randomized controlled trial. A group of students receives access to a structured learning tool designed to support independent study, while a comparison group continues with their standard study routine. Both groups are assessed regularly throughout the study period.
Our primary outcome of interest is performance on a standardized examination taken at the end of the study period. We also collect data on student engagement and learning behavior, as well as on classroom instruction. Where possible, we benchmark outcomes against students in urban areas.

Intervention

The intervention provides structured AI-assisted tutoring sessions to ninth-grade students in eight rural middle schools. Students assigned to the treatment group attend a daily two-hour session using an AI chatbot (Doubao) on tablets, following a fixed six-step structure: (1) a short pre-test on the day's topic, (2) AI-generated diagnostic feedback based on the pre-test, (3) an extended period of personalized AI-guided practice in which problem difficulty adapts to student performance, (4) an AI-generated summary of the key concepts covered, and (5) a post-test on the same topic. Students receive a small token incentive, redeemable for a gift, for completing all required steps of each session. Research assistants are present to monitor session integrity and record participation.
Students in the control group continue with their regular classroom instruction and complete the same pre- and post-tests as the treatment group, without access to the AI tutoring sessions, allowing learning gains to be compared across groups. The intervention is delivered over several weeks in the lead-up to the high school entrance examination (zhongkao), with schools entering the study on a staggered schedule."

Experimental design

This is an individually randomized controlled trial conducted in eight rural middle schools among ninth-grade students preparing for the zhongkao (high school entrance examination). The intervention takes place during the evening self-study period, a block of time with no formal classroom instruction, during which students are otherwise expected to study independently. Within each school, students are randomly assigned to treatment or control, stratified by baseline test score tertile and gender. Students assigned to treatment use this self-study time for a structured two-hour AI-assisted tutoring session with an AI chatbot on tablets, delivered by research staff. Students assigned to control spend this same evening self-study period as they normally would: studying independently, with a teacher available in the room to answer questions if a student asks, but without any structured instruction or AI access. Both groups complete the same periodic assessments. Schools enter the study on a staggered schedule between mid-April and mid-May 2026, so students accumulate varying amounts of exposure time before the exam. The main comparison of interest is between treatment and control students' performance on the zhongkao. A separate, non-randomized comparison sample of students in two urban schools was also surveyed to help interpret rural-urban differences in outcomes and AI access, but is not part of the randomized comparison.

Design details

Randomization method
Randomization done in office by computer (using a random number generator in statistical software), at the individual student level within each school, stratified by baseline academic performance and gender.
Randomization unit
Individual student, randomized within school. Randomization was implemented separately within each of the 8 participating rural schools (stratified by baseline academic performance tertile and gender within each school), rather than pooled across schools, so school functions as a stratification block, and the unit of random assignment is the individual student.
Clusters
Not applicable. This is an individually randomized design, not a clustered design. The 8 schools serve as stratification blocks (randomization occurs within each school), not as randomization clusters.
Observations
880 ninth-grade students (approximately 410 assigned to treatment and 470 assigned to control) across 8 rural middle schools. This reflects 940 students who completed the baseline survey, of whom 60 declined to participate in the experiment; the remaining 880 were randomized. The treatment/control split is not 50/50 because the number of treatment slots was constrained by the number of tablets purchased for the program (a budget constraint).
Arms
410 students in treatment group across 8 schools, stratified by baseline academic performance and gender.
Primary outcomes
The primary outcome is student performance on the zhongkao (中考), the high school entrance examination, measured as total score and, where available, subject-specific scores (particularly math), drawn from official exam/administrative records matched to each student. Where actual zhongkao answer sheets or item-level scoring can be obtained, we will additionally conduct a topic-level (question-by-question) analysis: for each exam question, we will code whether it falls within a topic covered during the AI tutoring sessions, and compare treatment-control differences in correctness on covered-topic questions versus non-covered-topic questions. This tests whether any overall treatment effect on total score is concentrated in the specific content areas addressed by the intervention, as opposed to a general test-taking or motivation effect.
Secondary outcomes
Short term outcomes include session pre/post test score gain, mock zhongkao test scores. Non-cognitive and psychological measures at endline: growth mindset beliefs about intelligence, math self-efficacy/persistence on difficult problems, perceived returns to effort, exam-related stress, and depressive-symptom items (sad, unhappy, hopeless feelings over the past 7 days). Educational aspirations and post-exam track choice: whether students plan to attend academic high school, vocational school, or enter work, and whether exposure to the program's instructors increased aspirations for further schooling. AI usage patterns and attitudes among all students (including control): frequency and purpose of independent AI use, beliefs about AI's helpfulness and dependency risk, and peer-to-peer spillover of AI-learned methods (whether students shared or learned techniques from classmates). Classroom environment spillovers: reported changes in classroom disruption (talking, phone use) during the intervention period, as a proxy for externalities on non-participating peers. Program satisfaction and take-up among treated students: perceived usefulness of each protocol step, self-reported attendance/compliance, and suggestions for improvement.

Cited by

No citation data available for this registration yet.

Keywords