Can a Tutor AI improve learning and other educational outcomes in post-secondary education?

How to cite

Higa, Minoru; Mouganie, Pierre. 2025. "Can a Tutor AI improve learning and other educational outcomes in post-secondary education?." AEA RCT Registry. AEARCTR-0016639. https://doi.org/10.1257/rct.16639-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Minoru Higa* , Pierre Mouganie

* Primary investigator

Registry ID
AEARCTR-0016639
DOI
https://doi.org/10.1257/rct.16639-1.0
First registered
Last updated
Status
In development
Trial dates
2025-09-22 – 2026-05-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/16639

Abstract

This trial evaluates the impact of providing access to a Tutor AI on learning and related educational outcomes in undergraduate programming courses in Colombia. The intervention consists of two treatment arms. In the Tutor AI Only arm, students receive access to an AI tutor for asynchronous academic activities such as homework, self-study, and projects. In the Tutor AI Plus arm, students receive the same asynchronous access, in addition to periodic in-class workshops where they solve exercises with AI support under the supervision of the instructor. Randomization occurs at the section level. Primary outcomes include academic performance (grades), absenteeism, course withdrawal, and hiring external tutors. Secondary outcomes include self-reported measures of confidence, motivation, and perceived usefulness of programming for the future. Data sources include baseline and follow-up surveys, focus groups, and administrative academic records.

Experimental design

In the pilot phase, course sections were randomly assigned to one of two treatment arms:
Tutor AI Only – Students in this group receive access to the AI tutor as an external assistant for asynchronous academic activities, including homework, self-study, and course projects.
Tutor AI Plus – Students in this group receive the same access to the AI tutor for asynchronous activities as in the Tutor AI Only arm, but additionally participate in periodic in-class workshops throughout the semester. In these workshops, students solve exercises with the support of the AI tutor under the guidance of the course instructor.
Prior to granting access to the AI tutor, the principal investigators (PIs) administer a baseline survey to measure students’ prior experience with AI, as well as their self-reported confidence, motivation, and perceived usefulness of learning programming.
At the end of the first semester, the PIs administer a follow-up survey to collect information on students’ experiences using the AI tutor and to reassess their confidence, motivation, and perceptions of the usefulness of learning programming.
To complement the quantitative data, the PIs also conduct a focus group with students at the end of the semester. Additionally, administrative records are collected to obtain students’ sociodemographic characteristics and academic information.

Design details

Randomization method
Randomization performed in the office by computer
Randomization unit
section
Clusters
First semester (pilot): 8 sections Instructor 1: 2 sections control Instructor 2: 3 sections, 1 treated with Tutor AI Only, 1 treated with Tutor AI Plus, 1 control Instructor 3: 3 sections, 1 treated with Tutor AI Only, 1 treated with Tutor AI Plus, 1 control Given the relatively small number of clusters, we will implement the wild cluster bootstrap-t procedure to obtain valid inference.
Observations
First semester (pilot): 364 students Second semester: 360 students Total: 724 students
Arms
Tutor AI Only: 96 students Tutor AI Plus: 98 students Control: 170 students
Primary outcomes
Grades Absenteeism Course Withdrawal/Drop out Hiring external tutors Self-reported confidence to learn programming Self-reported motivation to learn programming Self-reported perception that learning programming today will be useful for the future.

Cited by

No citation data available for this registration yet.

Keywords