It Takes a Village (Plus AI): A Pilot Study on AI-Driven Math Tutoring to Revolutionize Learning

How to cite

Thomas, Danielle; Koedinger, Kenneth R.. 2024. "It Takes a Village (Plus AI): A Pilot Study on AI-Driven Math Tutoring to Revolutionize Learning." AEA RCT Registry. AEARCTR-0014505. https://doi.org/10.1257/rct.14505-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Danielle Thomas* , Kenneth R. Koedinger

* Primary investigator

Registry ID
AEARCTR-0014505
DOI
https://doi.org/10.1257/rct.14505-2.0
First registered
Last updated
Status
In development
Trial dates
2024-04-01 – 2025-08-01
Countries
United States of America (Pennsylvania, West Virginia)
Registry record
http://www.socialscienceregistry.org/trials/14505

Abstract

Personalized Learning Squared (PLUS) is an edtech platform that uses a hybrid human-AI tutoring intervention to boost learning gains for middle school students from historically underserved communities. We know that tutoring works to improve learning, especially among students who are furthest behind. However, tutoring is a fairly resource intensive and expensive intervention. PLUS connects remote, human tutors to students working on math software in classrooms, and uses insights from the math software to identify needs in real time and help tutors allocate their attention between students. This optimizes student gains and provides a scalable solution to increase math learning. PLUS has been operating in classrooms since the 2022-2023 school year. They have been expanding into new schools and districts, improving the PLUS platform as well as their approach to onboarding and supporting teachers and tutors, and evaluating the impact of the PLUS platform iteratively. They plan to conduct a pilot study during the 2024-2025 school year that will continue to test the implementation of the PLUS program as well as the impact on student learning outcomes.

Intervention

PLUS Tutoring is a math tutoring program that operates within school hours during regular math classes and is designed to provide personalized tutoring for all students in a school. Using data from schools’ existing math practice software (Mathia, iReady, IXL, etc.), the PLUS Tutoring software helps allocate appropriate levels of tutoring to different students. Students in PLUS will receive one of three forms of PLUS tutoring: AI/ed tech tutor added (no human tutoring), on-demand student-initiated tutoring, or data-driven tutor-initiated tutoring. All students in the study will, one day per week during math class, use a device (typically a laptop) with educational technology systems, and open a video-conferencing platform (such as Zoom or Pencil Spaces). Treatment students will receive data-driven tutor-initiated tutoring, with tutors assigned to 2-6 students based on student needs and tutoring skills. Tutors will lead sessions between 2 and 20 minutes to build relationships, identify and address motivational challenges and help get students back on track to making progress. Tutors use a dashboard (PLUS Toolkit) to track student math achievement, practice effort, and prior progress. Comparison group students will have access to on-demand student-initiatives tutoring, with 1 tutor available to serve 10-15 students. During math classes, students may use video-conferencing platforms to request help from a remote tutor. These remote tutoring sessions are also predicted to last roughly 2-20 minutes.

Experimental design

We will use a regression discontinuity design (RD) to measure the impact of tutor-initiated versus student-initiated tutoring on math learning outcomes among middle school students. Previous test scores will be used as the forcing variable and a score cutoff will determine allocation into tutor-initiated or student-initiated tutoring. Students whose school records do not include this score will be randomly assigned. Spring 2024 PSSA/WVGSA scores will be used as the forcing variable for the RD. Anyone below the median PSSA score for a particular district, school, and grade, will be assigned to tutor-initiated tutoring, while students who score above the median will be assigned to student-initiated tutoring. Students with missing scores will be randomly assigned to one of the two groups. Additionally, we will analyze differences in outcomes across key subgroups, including historically marginalized students, students from low-income backgrounds, students with disabilities, English Language learners, and students from underserved populations. This will allow us to explore how these targeted groups respond to different tutoring models, shedding light on any variations in impact.

Design details

Randomization method
We will conduct a student-level randomization for students with missing historical PSSA and WVGSA data (done in office by a computer).
Randomization unit
student / individual
Clusters
0- no clusters
Observations
We anticipate around 3,000 students will participate in this study during the 2024-2025 school year. We anticipate around ⅔ will have test data from the previous year and will be involved in the RD, and the remaining ⅓ will have missing data and will be randomized.
Arms
N/A
Primary outcomes
Standardized end of year state math assessments (Pennsylvania System of School Assessment-PSSA and the West Virginia General Summative Assessment-WVGSA)
Secondary outcomes
We will also measure program and student progress in math using district-approved diagnostic assessments (e.g., i-Ready Diagnostic, Pennsylvania Classroom Diagnostic Tools (CDT), NWEA MAP, Star Assessment Diagnostic) and attendance.

Cited by

No citation data available for this registration yet.

Keywords