The Effects of Generative AI on Learning

How to cite

Irmert, Natalie; Franco, Catalina; Isaksson, Siri. 2024. "The Effects of Generative AI on Learning." AEA RCT Registry. AEARCTR-0014930. https://doi.org/10.1257/rct.14930-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Natalie Irmert* , Catalina Franco , Siri Isaksson

* Primary investigator

Registry ID
AEARCTR-0014930
DOI
https://doi.org/10.1257/rct.14930-2.0
First registered
Last updated
Status
In development
Trial dates
2024-12-03 – 2025-12-31
Countries
Private
JEL codes
I21, I24, J16
Registry record
http://www.socialscienceregistry.org/trials/14930

Abstract

Artificial Intelligence (AI) is becoming an increasingly important skill in the labor market, but will it also affect academic success? Recent research shows that current students --who will be facing this rapidly changing labor market-- are adopting AI tools at differential rates based on both gender and ability. Whether AI will affect adopters’ academic success hinges on whether AI interferes with or enhances learning, which in turn depends on whether AI is being used as a substitute for or complement of own effort. If AI harms learning, students with high adoption rates would be worse prepared for the labor market than those with low adoption rates. If AI enhances learning, students who do not become proficient at AI would be left behind. To assess the impact of AI on learning, we run a controlled lab experiment which allows us to restrict and allow access to AI in different between-subject treatment variations in which students learn about a new topic. We explore whether AI is employed in a way that causally creates a gap in learning and as a consequence payoffs. Our results provide evidence on the important question of whether the documented differential adoption and use of AI by gender and ability is likely to create gender gaps in academic success. In addition, we explore several mechanisms such as over-reliance on generative AI tools, engagement and motivation.

Intervention

We design a laboratory experiment with three between subject treatment variations. In the baseline treatment, students learn without access to AI. In the AI-assisted treatment, students learn with access to AI. In the AI-guided treatment, students learn with access to AI and examples of ways to interact with AI to promote learning. This will allow us to answer the crucial question of whether, and for whom, AI is helpful as a learning tool and whether providing guidance of how to use the tool matters.

Experimental design

Regardless of treatment, subjects will learn about the same topic. The treatment differences lies in whether or not they have access to ChatGPT/ChatGPT with guidance while learning.
Each session will adhere to the same procedure:
-Subjects learn for 15 minutes using written learning materials provided by the researchers.
-Subjects complete practice questions with access to different learning aids depending on treatment.
-Subjects take a 10 minute test without access to any learning aids.
-Subjects take a post-study survey asking them about demographics as well as attitudes and beliefs about AI.

Design details

Randomization method
Students draw number from a ballot upon arrival in the lab.
Randomization unit
individual
Clusters
no clusters
Observations
600 students
Arms
200 students control, 200 students AI assisted, 200 students AI guided
Primary outcomes
The main outcome variable will be the score that students get on their final exam in stage 3. We will compare across treatments to see whether access to (guided) AI helps or hurts learning outcomes, as measured by the stage 3 score.
Secondary outcomes
Subgroup analysis by gender, GPA. Secondary outcomes: being a top/bottom scorer, number of practice questions solved, number of practice questions solved correctly, time spent on exam, time spent on practice questions, usage of AI tool, usage of Google Search and Google Translate, number of prompts in ChatGPT, text analysis of prompts in ChatGPT, self-assessed learning, enjoyment of learning, being comfortable with tools, being confident about using tools, effort put in learning process, engagement with learning process, perceptions on over-reliance on tools.

Cited by

No citation data available for this registration yet.

Keywords