AI-Powered Tutoring: Unleashing the Full Potential of Personalized Learning with Khanmigo
How to cite
Oreopoulos, Philip. 2024. "AI-Powered Tutoring: Unleashing the Full Potential of Personalized Learning with Khanmigo." AEA RCT Registry. AEARCTR-0013519. https://doi.org/10.1257/rct.13519-2.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Tutoring stands out as a highly effective educational policy for improving student outcomes, but its implementation is hindered by issues of scalability and cost. One solution involves equipping teachers with enhanced skills in utilizing Computer Assisted Learning (CAL) to simulate the tutoring experience at a lower expense. A recent evaluation of this approach, called "Khoaching with Khan Academy" demonstrated meaningful improvements in standardized math test scores for elementary students whose teachers were randomly assigned to receive such assistance, as compared to students whose teachers did not benefit from it. Although this intervention successfully enhanced average math scores, the study also highlighted significant variations in CAL practice time both within and across classrooms receiving the treatment. To improve engagement of the Khan Academy platform and increase practice time, the KWiK program will leverage Khanmigo, a ChatGPT virtual assistant developed by Khan Academy, as a personalized tutor to support student’s progress with Khan Academy assignments. The additional improvement to the program is from dedicated practice time. Participating schools will set aside 30-45 minutes at the end of school during independent learning time for students to practice on KA with khanmigo.
We will conduct a grade-level RCT, randomizing Grades 6-8 within each of about 20 schools for students to receive the program for Tier 2 and 3 students. Teachers would receive mandatory training before the beginning of school with simplified instructions and a project manager within the district would be hired to manage khoach meetings and field visits to help ensure fidelity. All students would be given khanmigo access to maximize power for estimating the full program. The analysis will include impacts both at the end of the program in year 1, and a year after the program has concluded, at the end of year 2.
Intervention
Educational disparities rooted in socioeconomic status (SES) continue to be a major public policy issue, with significant impacts on long-term economic opportunities for young people. Traditional classroom settings, characterized by a wide range of academic levels among students, present substantial challenges to effective teaching and learning. Teachers cannot easily navigate diverse individual educational needs while being required to present a fixed set of topics over a limited period of time. Variance in academic preparedness and learning comprehension means that advancing new material can leave some students struggling to keep up, not due to a lack of capability but because of increasing foundational gaps. This heterogeneity is further exacerbated by factors such as the COVID-19 pandemic, which has widened the educational gaps and underscored the urgency of finding scalable solutions to provide personalized instruction.
Mastery learning, a pedagogical approach where students must demonstrate a high level of understanding before moving to the next topic, offers a framework for addressing these disparities. However, the traditional model of one-on-one tutoring, while effective, may be too costly and operationally challenging to scale. This project aims to explore the roles that Computer Assisted Learning (CAL) and Large Language Model (LLM) technologies might play as scalable solutions to these challenges to support mastery learning across diverse educational settings.
The main intervention is a program to support regular CAL practice for 30-45 minutes a day during a school's "What I Need" time at the end of each school day for Tier 2 and 3 Grade 6-6 students. Randomization is at the grade level across approximatley 50 schools. Schools receive training, support, and weekly guidance around successful implementation, with a focus on ensuring high dosage practice and adequate progress. Students follow an incremental roadmap of mastery learning both for math and English, with teachers and supervisors choosing particular units and topics for students to follow. Coaches help teachers and administrative staff ensure fidelity.
Experimental design
Grade 6-8 students classified as Tier 3 at the beginning of school similar Tier 2 students that schools that choose to include in the program would be the target population. Grades/schools would be randomized to determine which grades will receive the program and support, and which ones will be selected as comparison grades. Each school would have at least one grade selected, and not more than two. Grades selected for randomization will have their dedicated daily "What-I-Need" time used for Khan Academy practice both for math and ELA. Teachers would receive mandatory training before the beginning of school with simplified instructions and a project manager would be hired to manage khoach meetings and field visits to help ensure fidelity. Supervisors during WIN time will meet regularly (weekly) with khoaches or RTI leads to assess progress. All students would be given khanmigo access to maximize power for estimating the full program. The analysis will include impacts both at the end of the program in year 1, and a year after the program has concluded, at the end of year 2.
Design details
Cited by
No citation data available for this registration yet.