Humans versus Smartbots: Scaling-up behavioral interventions to reduce teacher shortages

How to cite

Ajzenman, Nicolas; Elacqua, Gregory; Hermosilla, Catalina; Nuñez, Graciela Perez; Mijares, Ana Teresa del Toro. 2024. "Humans versus Smartbots: Scaling-up behavioral interventions to reduce teacher shortages." AEA RCT Registry. AEARCTR-0014942. https://doi.org/10.1257/rct.14942-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Nicolas Ajzenman* , Gregory Elacqua , Catalina Hermosilla , Graciela Perez Nuñez , Ana Teresa del Toro Mijares

* Primary investigator

Registry ID
AEARCTR-0014942
DOI
https://doi.org/10.1257/rct.14942-1.0
First registered
Last updated
Status
On going
Trial dates
2024-11-01 – 2025-03-31
Countries
Chile (Entire country)
JEL codes
D91, I23, I25
Registry record
http://www.socialscienceregistry.org/trials/14942

Abstract

We design an experiment to assess the relative effectiveness of human mentors versus AI-powered chatbots in motivating students to pursue a teaching career in Chile. We work with Elige Educar, a Chilean NGO that implements a mentoring program for students interested in education or related fields. Participating high school students are randomly assigned to one of three groups: a group that re- ceives support from a human mentor via WhatsApp; a group that receives support from an AI-powered chatbot, also through WhatsApp; and a passive control group that receives no support. We assess the effectiveness of each treatment in motivating students to register for an education major through Chile’s centralized university admissions process. Beyond addressing Chile’s immediate teacher recruitment challenges, the study contributes to the emerging research on AI in education by empirically assessing the potential of generative AI to provide personalized, scalable mentoring.

Intervention

The intervention is a randomized controlled trial comparing two methods of mentoring high school students in Chile about teaching careers: human mentors and an AI-powered chatbot named "Kai." Working with the NGO Elige Educar, the study targets students who registered for Chile's university entrance exam (PAES) and expressed interest in education or social sciences careers.

Students are randomly assigned to receive WhatsApp-based mentoring from either trained human mentors (typically education students), an AI chatbot trained on Elige Educar's resources using ChatGPT-4mini, or no support (control group). The mentoring occurs from October 2024 to January 2025, with weekly activation of student subgroups. Both human mentors and the chatbot provide information about education programs, scholarships, career benefits, and application requirements. They make up to two contact attempts per student during their assigned intervention week.

A key innovation is expanding beyond students already interested in teaching to include those interested in social sciences more broadly. The intervention includes an intensive support phase during university applications in January 2025, when both treatment arms contact all non-opted-out students to provide application guidance. The study will evaluate effects on students' likelihood of applying to education programs through Chile's centralized university admissions process.

See PaP for more details.

Experimental design

The study is a randomized controlled trial evaluating the effectiveness of AI-powered chatbots versus human mentors in motivating Chilean high school students to pursue teaching careers. The experiment has three arms:

Human mentor treatment: Students receive WhatsApp mentoring from trained education students using Elige Educar's "Quiero Ser Profe" methodology.
AI chatbot treatment: Students interact with "Kai," an AI-powered chatbot trained on Elige Educar's resources using ChatGPT-4mini.
Control group: No contact or support.

The sample includes 43,539 students who registered for Chile's university entrance exam and expressed interest in education (54.4%) or social sciences (45.6%). Treatment assignment is stratified by gender, school type (public/private-subsidized/private), and initial career interests. The human mentoring arm includes 9,000 students, chatbot arm 20,000 students, and control group 14,539 students.
Both treatment arms operate October 2024-January 2025, making two contact attempts per student during their assigned week. An intensive support phase occurs during university applications (January 2024). The intervention uses Elige Educar's WhatsApp Business account to ensure source credibility.
Primary outcomes measure students' likelihood of applying to education programs through Chile's centralized university admissions system. Administrative data comes from DEMRE (Department of Evaluation, Measurement and Educational Registry).

Design details

Randomization method
Randomization done in office by a computer
Randomization unit
Student, no clustering.
Clusters
No clusters
Observations
Total sample size: 43,539 students Breakdown by arm: Human mentor treatment: 9,000 students AI chatbot treatment: 20,000 students Control group: 14,539 students Unit of randomization is individual students (no clustering).
Arms
Total sample size: 43,539 students Breakdown by arm: Human mentor treatment: 9,000 students AI chatbot treatment: 20,000 students Control group: 14,539 students Unit of randomization is individual students (no clustering).
Primary outcomes
The study has three primary outcomes measuring students' preferences for teaching careers: Whether a student ranks an education major as their first choice in the centralized university application system (main outcome) Proportion of education majors in a student's choice set Whether they applied to at least one education major Data for these outcomes comes from administrative records from DEMRE, Chile's university admissions authority.
Secondary outcomes
There is also one secondary outcome: actual enrollment in education programs. However, this outcome requires cautious interpretation since enrollment depends on a centralized assignment algorithm that considers all students' choices, meaning treatment effects could be biased by spillovers between treatment and control groups. Data for these outcomes comes from administrative records from DEMRE, Chile's university admissions authority.

Cited by

No citation data available for this registration yet.

Keywords