Adoption and Effectiveness of AI Tutoring in Tertiary Education

How to cite

Hermes, Henning; Isphording, Ingo. 2024. "Adoption and Effectiveness of AI Tutoring in Tertiary Education." AEA RCT Registry. AEARCTR-0014959. https://doi.org/10.1257/rct.14959-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Henning Hermes* , Ingo Isphording

* Primary investigator

Registry ID
AEARCTR-0014959
DOI
https://doi.org/10.1257/rct.14959-1.0
First registered
Last updated
Status
In development
Trial dates
2024-12-09 – 2026-12-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/14959

Abstract

We study whether and how much AI technologies improve learning and educational outcomes in a tertiary education context. We test this question using a large-scale randomized field experiment. Adapting an encouragement design, we evaluate which students select into usage of a targeted AI assistant within an online educational program, and assess the effects of this AI assistant on educational outcomes using data on study progress and grades based on admin data from the university.

Intervention

Our RCT investigates the effects of encouraging students to use an AI assistant on their study success. The AI assistant, integrated into the university's online learning platform, is a Chatbot that offers students support by answering clarifying questions, providing examples, looking up definitions, and referencing relevant study materials. The Chatbot, which is technically built on the current GPT-4 framework, has been trained with direct references to relevant sources such as study letters, further literature or presentations, in order to assist students in their individual learning process. Students in the intervention group will receive specific encouragement to use the Chatbot more intensively, while a control group will not receive such encouragement.

Experimental design

We will conduct a field experiment in which participants are randomly assigned to either the treatment or control group. Participants in the treatment group will receive encouragement to use an AI assistant (Chatbot), while those in the control group will not receive this encouragement. The AI assistant is generally available for all students. All students will be part of the experiment, either in the treatment group or the control group. The experiment will also include a baseline survey for all students to gather information on their perceptions and usage of the Chatbot, as well as personal characteristics. Main outcomes will be provided with administrative data from the university as well as an endline survey.

Design details

Randomization method
Treatment assignment by running odd/even contractual number at sign up.
Randomization unit
Individual student
Clusters
no clustering
Observations
7000 students
Arms
3500 treatment, 3500 control
Primary outcomes
First, we will analyze the usage of the AI assistant (Chatbot), selection into usage, and describe the type of usage. Specific focus will be given to selection based on initial levels of skills, in particular, digital skills. Second, we will evaluate effects of Chatbot use on educational success, measured as study progress (completed modules) and academic performance (grades).
Secondary outcomes
- Chatbot usage vs. classical tutor support - Heterogeneity analysis: Comparing usage of the Chatbot, study progress and academic performance by (i) gender, (ii) critical thinking and problem solving skills, (iii) digital literacy / competencies, (iv) socioeconomic background, and (v) usage type (see below). - Perceived Benefits: Assessed through survey questions regarding students' perceptions of the benefits and returns from using the Chatbot. - Satisfaction: Evaluated through surveys measuring satisfaction with the Chatbot.

Cited by

No citation data available for this registration yet.

Keywords