Complementing Teachers: Can Artificial Intelligence Improve Student Learning by Addressing Learning Variability?

How to cite

Morales, Juan Munoz; Barrera-Osorio, Felipe. 2026. "Complementing Teachers: Can Artificial Intelligence Improve Student Learning by Addressing Learning Variability?." AEA RCT Registry. AEARCTR-0017147. https://doi.org/10.1257/rct.17147-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Juan Munoz Morales* , Felipe Barrera-Osorio

* Primary investigator

Registry ID
AEARCTR-0017147
DOI
https://doi.org/10.1257/rct.17147-1.0
First registered
Last updated
Status
On going
Trial dates
2025-08-01 – 2027-12-31
Countries
Private
JEL codes
C93, I21, J16, J24, O15, O33
Registry record
http://www.socialscienceregistry.org/trials/17147

Abstract

Education systems in developing countries face a persistent learning crisis characterized by low average achievement and substantial disparities in student learning. This paper evaluates a teacher-level, artificial intelligence–enabled intervention designed to support lesson planning and promote within-classroom differentiated instruction. We implement a school-level randomized controlled trial in low-income public secondary schools in Bogotá, Colombia. The intervention provides mathematics teachers with access to an AI-based lesson-planning platform that integrates learner variability and growth-oriented pedagogical frameworks, reducing preparation costs while improving instructional alignment with heterogeneous student needs. We assess impacts on student learning, with particular attention to the heterogeneity of the effect in a context marked by pronounced educational inequality. Using baseline and endline student assessments and complementary teacher surveys, we estimate both intention-to-treat and treatment-on-the-treated effects, and examine treatment effect heterogeneity using pre-specified subgroup analyses and data-driven methods.

Intervention

In partnership with the Secretaría de Educación Distrital (SED)—the local education authority in Bogotá, Colombia—, and Mentu—a Colombian startup that provides AI-driven learning technologies—, we designed an intervention intended to enhance teachers’ value-added while addressing the heterogeneous learning needs of students from diverse backgrounds. The intervention targeted teachers working in low-income areas of the city and consisted of two components: (i) access to ShaIA, an AI-based pedagogical support platform designed to address learning variability; and (ii) a structured training program to support teachers in the effective use of the platform.

Access to AI-based pedagogical support: ShaIA is an AI-driven pedagogical ecosystem that provides personalized guidance to teachers. The platform is designed to support mathematics instruction by assisting teachers in planning and implementing lessons aligned with curricular standards and students’ diverse learning needs. Treated teachers were granted access to the platform, where they created a course profile—describing course characteristics to the algorithm—and specified learning objectives in mathematics.

ShaIA generates recommendations by channeling a large language model guided by two core pedagogical principles: learning variability and math mindsets.

1. Learning variability: ShaIA promotes inclusive teaching practices aligned with the prioritized learning objectives of the national curriculum. A central feature of the platform is its integration of the Learner Variability framework (leveraging Digital Promise’s Learner Variability Navigator), which tailors pedagogical recommendations based on models capturing students’ diverse backgrounds, needs, and learning profiles.

Teachers incorporate learning variability into ShaIA through a three-step process. First, they select a learning model from four available options: mathematics, language, adult learning, and 21st-century learners. While ShaIA provides an initial recommendation, teachers may modify the model to better align it with their instructional context and objectives. Second, teachers define a class profile by selecting relevant variability factors that characterize their students (e.g., presence of non–Spanish-speaking students, migrants, students with disabilities, or students affected by trauma, etc.). Based on this profile, ShaIA presents a list of 35 factors identified as relevant for mathematics learning.Teachers then select between 10 and 15 factors they consider most salient for their classroom. Third, teachers select inclusion strategies. For each chosen factor, ShaIA suggests evidence-based pedagogical strategies that enable teachers to address classroom diversity simultaneously.These strategies are automatically incorporated into the generation of lesson plans, activities, and instructional resources, ensuring that the platform’s outputs are aligned with the actual characteristics of the class.

2. Math mindsets. ShaIA's suggested pedagogical strategies are grounded in the Math Mindsets approach, which draws on robust recent evidence on effective mathematics teaching. This approach emphasizes understanding, creativity, and confidence, and is rooted in the concept of a growth mindset applied specifically to mathematics learning.

Building on this framework, ShaIA offers tools for lesson planning involving activity design, project-based learning, and development of formative assessments and grading rubrics. Its recommendations are informed by the selected learner variability factors and structured around five pedagogical tools of the math mindsets framework: mathematical experience, worksheets, number talks, pattern talks, and feedback practice. These tools are designed to strengthen mathematical communication, reasoning, problem solving, and modeling, while also supporting procedural fluency as an outcome of numerical flexibility and algebraic thinking rather than through isolated or mechanical practice.

Overall, ShaIA is designed to complement rather than replace teachers. By automating lesson planning and reducing its preparation time, the platform aims to increase teacher engagement and promote higher-value activities such as instructional reflection and individualized student support. By moving beyond a one-size-fits-all approach, ShaIA seeks to enable teachers, particularly those with limited access to pedagogical resources, to adapt instructional strategies to classrooms with heterogeneous learning profiles.

Training on the use of ShaIA: Effective use of the platform requires that teachers understand its functionality and perceive its pedagogical value. To promote adoption and sustained use, \textit{Mentu} and the SED implemented a structured training program for treated teachers. The program consisted of three school visits, three in-person workshops, and three one-hour webinars. The workshops were intended to familiarize teachers with the platform by modeling teaching strategies using ShaIA. The visits were intended to gather feedback and to review the teachers' work plan. Finally, the webinars were designed to reinforce ShaIA's instructional applications, and support its integration into classroom practice.

Experimental design

We evaluate the effects of providing teachers with AI support by offering school-level access to ShaIA. Treatment was assigned among 158 eligible schools in Bogotá, Colombia. Due to budget constraints, the agreement with the SED stipulated that teachers in lower secondary (grades 6th to 9th) in 58 schools would receive access to the platform. However, participation was voluntary and depended on each school principal’s willingness to join the evaluation. Schools originally assigned to treatment could therefore opt out. If they did, they neither received access to ShaIA nor participated in the experiment.

Because of the risk of non-participation, we implemented a block-randomization algorithm in which schools were grouped into blocks and treatment was assigned within each block. This ensured that, even if a school assigned to treatment chose not to participate in a given block, the randomization would remain valid across the other blocks of the experiment. The randomization proceeded in three stages:

1. For each eligible school, we created a covariate index defined as the average of several school-level characteristics.

2. Schools were then ranked by the covariate index and grouped into blocks. More similar schools were assigned to the same block.

3. Treatment was randomized within each block, yielding a treatment group of 70 schools and a control group of 88.

Due to the random nature of the treatment assignment, we can estimate the effect of providing teachers with AI support by comparing outcomes for teachers and students in treated schools with those in control schools. Because treatment was assigned using block randomization, these comparisons must be made within blocks; therefore, we condition on block indicators.

Some mathematics teachers in treated schools may choose not to participate in the intervention. As a result, some units assigned to treatment may remain untreated, implying that our strategy identifies an Intention-to-Treat (ITT) effect. To account for imperfect compliance, we additionally define an indicator variable equal to one if teacher adopts the technology. We then instrument treatment adoption with treatment assignment to estimate the Treatment-on-the-Treated (TOT) effect.

Internal Validity: The internal validity of the experiment relies on treatment being assigned orthogonally to baseline characteristics. To assess this, we will conduct balance tests using school-level administrative data and baseline survey data.

Heterogeneity: Treatment effects of educational interventions are known to be highly heterogeneous. This makes it essential to study effect heterogeneity when evaluating the impact of ShaIA. We approach this in two ways.

First, the context of Bogotá suggests substantial pre-existing gaps in math achievement by gender and migrant status. Prior evidence shows that girls in Colombia score significantly lower in math, a gap largely explained by within-school factors. Likewise, migrant students tend to learn less in non-adapted environments, and lower learning rates for migrant populations have been documented in Colombia as well. These patterns provide theoretical grounds to expect larger treatment effects for these groups, given that ShaIA aims specifically to address within-classroom heterogeneity. Therefore, we include these two heterogeneity dimensions in our pre-analysis plan as pre-specified heterogeneous effects that test our original hypothesis. In addition, given the nature of the treatment, we allow for the possibility that treatment effects may vary across teacher characteristics and levels of treatment exposure. Accordingly, we will examine treatment effect heterogeneity by teacher type—specifically age, baseline AI knowledge, and gender—as well as by treatment exposure (time exposed to ShaIA and timing of the workshops).

Second, we will formally test the null hypothesis that the standard deviation of treatment effects is equal to zero. Rejecting this null would indicate heterogeneous impacts, in which case we will estimate Conditional Average Treatment Effects (CATEs) based on pre-specified characteristics. This data-driven analysis will help identify the subgroups that benefit the most from the intervention. Our hypothesis is that girls and migrant students are likely to experience the largest gains, as the intervention is designed to reduce within-class learning gaps.

Design details

Randomization method
Because of the risk of non-participation, we implemented a block-randomization algorithm in which schools were grouped into blocks and treatment was assigned within each block. This ensured that, even if a school assigned to treatment chose not to participate in a given block, the randomization would remain valid across the other blocks of the experiment. The randomization proceeded in three stages: 1. For each eligible school, we created a covariate index defined as the average of several school-level characteristics.\footnote{All covariates were standardized and then averaged to build a one-dimensional index. 2. Schools were then ranked by the covariate index and grouped into 52 blocks. More similar schools were assigned to the same block. 3. Treatment was randomized within each block, yielding a treatment group of 70 schools and a control group of 88.
Randomization unit
Randomization was clustered at the school level, with 70 schools treated and 88 in the control group.
Clusters
The intervention includes 158 schools, out of which 55 are to be treated and the rest will be part of the control group.
Observations
Current funding allow us to sample 6,000 students and 270 teachers within 40 treated and 50 control schools. However, the final sample size depends is more funding is available in which case we will use it to sample a larger number of students.
Arms
55 schools will be assigned to the treatment group. The remaining 103 schools will be assigned to the control group.
Primary outcomes
Student learning and teacher engagement.

Cited by

No citation data available for this registration yet.

Keywords