Pre-Analysis Plan for A Large RCT of ALTER-Math: AI-Powered Learning by Teaching to Enhance and Renovate Middle School Math Learning

How to cite

Li, Chenglu; Xing, Wanli; Liu, Zifeng. 2026. "Pre-Analysis Plan for A Large RCT of ALTER-Math: AI-Powered Learning by Teaching to Enhance and Renovate Middle School Math Learning." AEA RCT Registry. AEARCTR-0019131. https://doi.org/10.1257/rct.19131-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Chenglu Li* , Wanli Xing , Zifeng Liu

* Primary investigator

Registry ID
AEARCTR-0019131
DOI
https://doi.org/10.1257/rct.19131-1.0
First registered
Last updated
Status
In development
Trial dates
2026-10-01 – 2027-03-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/19131

Abstract

This document pre-specifies our planned analyses of the evaluation of AI-augmented Learning by Teaching to Enhance and Renovate Math Learning (ALTER-Math), which aims to double the rate of algebra learning for middle school students, particularly for those from low-income households, by developing effective, scalable, and responsible AI-enhanced teachable agents, utilizing large language models (LLMs) and cutting-edge AI technologies. This intervention analyses to what extent an AI-powered teachable agent can enhance middle-school math learning by turning students into the AI’s teachers through a large-scale randomized controlled trial. The primary research question is: How effectively does the AI-powered teachable agent engage students and improve mathematical achievement? We hypothesize that students assigned to ALTER-Math will outperform controls on state math assessments, show higher interest in mathematics, and exhibit stronger engagement (time-on-task and persistence), with equitable outcomes for historically underserved students; we also expect effects to generalize beyond Math Nation contexts. The intervention embeds a multimodal AI teachable agent (students explain, correct, and guide the agent across text, tables, graphs, and drawings) into regular coursework, with teachers receiving PD and weekly implementation across a 28-week term. We will run a three-level cluster randomized trial at the school level with ~50 schools, ~150 teachers, and ~7,500 students (~50% low-income). Primary outcomes are performance on Florida’s FAST assessment (PM1 baseline to PM3 post), student interest in mathematics (Student Interest in Mathematics Scale), and log-based engagement indicators; subgroup analyses will test differential effects by FRPM, race/ethnicity, and EL status.

Intervention

Middle school math, a key milestone to engaging in advanced mathematical study, significantly influences students' STEM education, career trajectories, and the broader national STEM workforce (Kieran, 2004; Matthews & Farmer, 2008). However, at this moment, middle school mathematics education faces major challenges, including the lack of resources and approaches to engage students both cognitively and affectively (Major et al., 2021; Mitchall & Jaeger, 2018; Williams et al., 2018). Nationally, only about 26% of eighth graders are proficient in math, and in some high-poverty districts, the rates are in the single digits (Learning Engineering Virtual Institute, n.d.)​. Such disparities highlight a critical equity issue: without effective interventions, many historically underserved students fall behind in math, limiting their future educational and economic opportunities. Therefore, improving middle school math outcomes is a priority for policymakers focused on closing achievement gaps and promoting workforce readiness. Findings from a recent study at the National Bureau of Economic Research (Guryan et al., 2021) offer a promising solution, where the research demonstrates that individualized tutoring significantly improved annual math learning gains, improved test scores, and reduced course failure rates for historically underserved students. This rigorous research provides stakeholders with an empirically and longitudinally evidenced strategy to support students, especially those from an economically disadvantaged background, by catering to their personalized needs and individual differences.
Augmenting mathematics tutoring with artificial intelligence (AI) tools has demonstrated potential for success. AI presents scalable solutions for students with diverse learning needs through adaptive learning (e.g., DreamBox Learning Inc.), intelligent tutoring systems (e.g., MATHia, formerly known as Cognitive Tutor, Ritter et al., 2007), and teacher-centered AI (e.g., ASSISTments, Heffernan & Heffernan, 2014). Recently, AI technologies have advanced from structured, expert-modeled, and rule-based systems to more dynamic, intuitive, and context-aware methodologies through generative artificial intelligence such as large language models (LLMs). Despite AI's potential for reimagining mathematics education, current implementations often treat students as passive recipients of AI-driven decisions, where AI provides direct answers, guidance, and feedback to students. Supported by the Learning Engineering Virtual Institute (LEVI), our project, AI-powered Learning by Teaching to Enhance and Renovate Math Learning (ALTER-Math), addresses these gaps by transforming students from passive learners into proactive teachers of AI-powered teachable agents, utilizing LLMs and multimodal AI technologies. While AI is yet to be a superior teacher, it is already an excellent student.
Led by the University of Miami (UM) and the University of Utah (Utah), with a collaborative network of experts and engineers from Stanford, Duke, Vanderbilt, and Math Nation, ALTER-Math aims to enhance math learning for middle school students, especially those from low-income households. In ALTER-Math, students guide, teach, and interact with the AI agent in multiple representations (e.g., graphing, drawing, tables, and texts) to solve math problems embedded into different real-world scenarios (e.g., music, environmental issues, and cultural festivals). Through partnership with and direct integration into Math Nation—an online math curriculum with over one million active students annually—we have developed robust cyberinfrastructure guided by effective learning sciences principles, conducted classroom studies showing promising results (Xing et al., 2025), and gained key insights to refine our AI-driven approach.

Experimental design

To rigorously evaluate the efficacy and scalability of ALTER-Math, we propose a three-level cluster randomized trial (CRT) involving approximately 7,500 students (~50% low-income) and 150 teachers from 53 schools in Florida. Our study will be guided by the overarching research question of How effectively does the AI-powered teachable agent engage students and enhance students’ mathematical achievements? We will also conduct sub-group analysis to answer research questions, including (1) To what extent does the use of ALTER-Math within and outside Math Nation impact students’ mathematical engagement and achievements? (2) How do usage patterns differ between historically marginalized middle school students regarding race, socio-economic status, and English learner status, and their more advantaged peers, when using the AI teachable agent? and (3) To what extent does the AI teachable agent influence math learning outcomes for historically marginalized middle school students compared to their more advantaged peers?

Study Design Overview: The 28-week study (October 2026–April 2027) will be a three-level cluster randomized trial with students nested within teachers nested within schools. Random assignment to treatment and control conditions will occur at the school level. We propose to include 50 middle schools (25 per condition), with an average of 3 teachers per school. Each teacher is expected to teach two classrooms of approximately 25 students, resulting in an estimated total sample size of 7,500 students. To strengthen contrasts, we will include students from Duval County, who do not have access to Math Nation. This allows us to examine the effects of ALTER-Math in a distinct context where Math Nation is not available. For schools with access to Math Nation, we will leverage the platform to randomize access to ALTER-Math. This approach will streamline teacher recruitment and simplify the management of treatment and control groups. For Duval County, we will recruit teachers during in-person professional development (PD) sessions, document their intent to participate, and randomly assign schools to intervention or control conditions. By randomizing at the school level, we minimize potential spillover effects that could occur with student-level randomization and ensure compliance with What Works Clearinghouse (WWC) Standards 5.0 without reservations (U.S. Department of Education et al., 2022). While baseline equivalence on pre-test measures is generally expected in large randomized samples, differences greater than 0.05 but less than 0.25 standard deviations will be addressed by including pre-test measures as covariates in our analytic models.

Design details

Randomization method
Randomization will occur at the school level after teacher recruitment and baseline rosters are finalized. Schools will be assigned in a 1:1 ratio to treatment or business-as-usual control using a reproducible randomization script, with assignment conducted by the research team before implementation begins. To support balance across study conditions while keeping the design transparent, randomization will be stratified by district and Math Nation access context. Within each stratum, schools will be randomly assigned to treatment or control to the extent possible, recognizing that the number of participating schools may vary across districts and strata. After randomization, the research team will examine baseline equivalence across conditions using available student-level characteristics, including gender, FRPM eligibility, and other relevant demographic indicators. If any pre-specified baseline covariate differs across arms by more than 0.25 standard deviations in the provisional draw, we will use restricted randomization or rerandomization before assignments are released. Students and teachers inherit the assignment of their school.
Randomization unit
School level randomization
Clusters
50 schools, with each school on average having 3 teachers
Observations
7,500 students, 150 teachers, and 50 schools
Arms
25 schools control, and 25 schools treatment
Primary outcomes
The Florida Assessment of Student Thinking (FAST) will serve as a distal measure of middle school students' mathematics performance. Administered three times a year and aligned with Florida’s BEST standards, FAST provides a comprehensive, computer-adaptive assessment of student growth, with tests lasting 100–120 minutes and featuring multiple-item formats (Florida Department of Education, 2024). We will use the first wave of FAST (PM1) in Fall 2026 as the baseline and the third wave (PM3) in Spring 2027 as the post-intervention measure to evaluate the effects of ALTER-Math.
Secondary outcomes
To assess students’ interest in mathematics, we will use the Student Interest in Mathematics Scale (Wininger et al., 2014), a validated 17-item Likert-scale instrument measuring emotion, value, knowledge, and engagement (Cronbach's α = 0.76–0.87). Lastly, students’ interactive log data will be used as a proxy for engagement, capturing time-on-task, retention, activity frequency, and interactions with AI agents (Kim et al., 2020). Engagement trends will be analyzed weekly across the RCT period to assess growth

Cited by

No citation data available for this registration yet.

Keywords