Pre-Analysis Plan for A Large RCT of ALTER-Math: AI-Powered Learning by Teaching to Enhance and Renovate Middle School Math Learning
How to cite
Li, Chenglu; Xing, Wanli; Liu, Zifeng. 2026. "Pre-Analysis Plan for A Large RCT of ALTER-Math: AI-Powered Learning by Teaching to Enhance and Renovate Middle School Math Learning." AEA RCT Registry. AEARCTR-0019131. https://doi.org/10.1257/rct.19131-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This document pre-specifies our planned analyses of the evaluation of AI-augmented Learning by Teaching to Enhance and Renovate Math Learning (ALTER-Math), which aims to double the rate of algebra learning for middle school students, particularly for those from low-income households, by developing effective, scalable, and responsible AI-enhanced teachable agents, utilizing large language models (LLMs) and cutting-edge AI technologies. This intervention analyses to what extent an AI-powered teachable agent can enhance middle-school math learning by turning students into the AI’s teachers through a large-scale randomized controlled trial. The primary research question is: How effectively does the AI-powered teachable agent engage students and improve mathematical achievement? We hypothesize that students assigned to ALTER-Math will outperform controls on state math assessments, show higher interest in mathematics, and exhibit stronger engagement (time-on-task and persistence), with equitable outcomes for historically underserved students; we also expect effects to generalize beyond Math Nation contexts. The intervention embeds a multimodal AI teachable agent (students explain, correct, and guide the agent across text, tables, graphs, and drawings) into regular coursework, with teachers receiving PD and weekly implementation across a 28-week term. We will run a three-level cluster randomized trial at the school level with ~50 schools, ~150 teachers, and ~7,500 students (~50% low-income). Primary outcomes are performance on Florida’s FAST assessment (PM1 baseline to PM3 post), student interest in mathematics (Student Interest in Mathematics Scale), and log-based engagement indicators; subgroup analyses will test differential effects by FRPM, race/ethnicity, and EL status.
Intervention
Middle school math, a key milestone to engaging in advanced mathematical study, significantly influences students' STEM education, career trajectories, and the broader national STEM workforce (Kieran, 2004; Matthews & Farmer, 2008). However, at this moment, middle school mathematics education faces major challenges, including the lack of resources and approaches to engage students both cognitively and affectively (Major et al., 2021; Mitchall & Jaeger, 2018; Williams et al., 2018). Nationally, only about 26% of eighth graders are proficient in math, and in some high-poverty districts, the rates are in the single digits (Learning Engineering Virtual Institute, n.d.). Such disparities highlight a critical equity issue: without effective interventions, many historically underserved students fall behind in math, limiting their future educational and economic opportunities. Therefore, improving middle school math outcomes is a priority for policymakers focused on closing achievement gaps and promoting workforce readiness. Findings from a recent study at the National Bureau of Economic Research (Guryan et al., 2021) offer a promising solution, where the research demonstrates that individualized tutoring significantly improved annual math learning gains, improved test scores, and reduced course failure rates for historically underserved students. This rigorous research provides stakeholders with an empirically and longitudinally evidenced strategy to support students, especially those from an economically disadvantaged background, by catering to their personalized needs and individual differences.
Augmenting mathematics tutoring with artificial intelligence (AI) tools has demonstrated potential for success. AI presents scalable solutions for students with diverse learning needs through adaptive learning (e.g., DreamBox Learning Inc.), intelligent tutoring systems (e.g., MATHia, formerly known as Cognitive Tutor, Ritter et al., 2007), and teacher-centered AI (e.g., ASSISTments, Heffernan & Heffernan, 2014). Recently, AI technologies have advanced from structured, expert-modeled, and rule-based systems to more dynamic, intuitive, and context-aware methodologies through generative artificial intelligence such as large language models (LLMs). Despite AI's potential for reimagining mathematics education, current implementations often treat students as passive recipients of AI-driven decisions, where AI provides direct answers, guidance, and feedback to students. Supported by the Learning Engineering Virtual Institute (LEVI), our project, AI-powered Learning by Teaching to Enhance and Renovate Math Learning (ALTER-Math), addresses these gaps by transforming students from passive learners into proactive teachers of AI-powered teachable agents, utilizing LLMs and multimodal AI technologies. While AI is yet to be a superior teacher, it is already an excellent student.
Led by the University of Miami (UM) and the University of Utah (Utah), with a collaborative network of experts and engineers from Stanford, Duke, Vanderbilt, and Math Nation, ALTER-Math aims to enhance math learning for middle school students, especially those from low-income households. In ALTER-Math, students guide, teach, and interact with the AI agent in multiple representations (e.g., graphing, drawing, tables, and texts) to solve math problems embedded into different real-world scenarios (e.g., music, environmental issues, and cultural festivals). Through partnership with and direct integration into Math Nation—an online math curriculum with over one million active students annually—we have developed robust cyberinfrastructure guided by effective learning sciences principles, conducted classroom studies showing promising results (Xing et al., 2025), and gained key insights to refine our AI-driven approach.
Experimental design
To rigorously evaluate the efficacy and scalability of ALTER-Math, we propose a three-level cluster randomized trial (CRT) involving approximately 7,500 students (~50% low-income) and 150 teachers from 53 schools in Florida. Our study will be guided by the overarching research question of How effectively does the AI-powered teachable agent engage students and enhance students’ mathematical achievements? We will also conduct sub-group analysis to answer research questions, including (1) To what extent does the use of ALTER-Math within and outside Math Nation impact students’ mathematical engagement and achievements? (2) How do usage patterns differ between historically marginalized middle school students regarding race, socio-economic status, and English learner status, and their more advantaged peers, when using the AI teachable agent? and (3) To what extent does the AI teachable agent influence math learning outcomes for historically marginalized middle school students compared to their more advantaged peers?
Study Design Overview: The 28-week study (October 2026–April 2027) will be a three-level cluster randomized trial with students nested within teachers nested within schools. Random assignment to treatment and control conditions will occur at the school level. We propose to include 50 middle schools (25 per condition), with an average of 3 teachers per school. Each teacher is expected to teach two classrooms of approximately 25 students, resulting in an estimated total sample size of 7,500 students. To strengthen contrasts, we will include students from Duval County, who do not have access to Math Nation. This allows us to examine the effects of ALTER-Math in a distinct context where Math Nation is not available. For schools with access to Math Nation, we will leverage the platform to randomize access to ALTER-Math. This approach will streamline teacher recruitment and simplify the management of treatment and control groups. For Duval County, we will recruit teachers during in-person professional development (PD) sessions, document their intent to participate, and randomly assign schools to intervention or control conditions. By randomizing at the school level, we minimize potential spillover effects that could occur with student-level randomization and ensure compliance with What Works Clearinghouse (WWC) Standards 5.0 without reservations (U.S. Department of Education et al., 2022). While baseline equivalence on pre-test measures is generally expected in large randomized samples, differences greater than 0.05 but less than 0.25 standard deviations will be addressed by including pre-test measures as covariates in our analytic models.
Design details
Cited by
No citation data available for this registration yet.