Experimental Evidence on the Learning Impact of Generative AI

How to cite

Reyes, Germán; Contractor, Zara. 2025. "Experimental Evidence on the Learning Impact of Generative AI." AEA RCT Registry. AEARCTR-0015806. https://doi.org/10.1257/rct.15806-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Germán Reyes* , Zara Contractor

* Primary investigator

Registry ID
AEARCTR-0015806
DOI
https://doi.org/10.1257/rct.15806-1.1
First registered
Last updated
Status
Completed
Trial dates
2025-04-21 – 2025-05-20
Countries
United States of America (Vermont)
JEL codes
I20, I23, J24, O33
Registry record
http://www.socialscienceregistry.org/trials/15806

Abstract

This study examines how access to generative artificial intelligence (AI) tools affects student learning outcomes in higher education. Using a randomized controlled experiment, we assign undergraduate students to complete learning tasks on unfamiliar academic topics either with or without access to generative AI tools. The experiment consists of two sessions approximately one week apart, allowing us to measure both immediate performance and longer-term knowledge retention. In the first session, participants complete a baseline assessment, engage in a structured learning exercise on an assigned topic, and produce a written analysis. Only participants in the treatment group are permitted to use generative AI during this process. In the second session, all participants complete knowledge retention assessments without access to AI. By comparing performance across conditions, we aim to identify causal effects of AI access on learning outcomes and explore potential mechanisms behind these effects. The findings will provide empirical evidence on an important question facing higher education as these technologies become increasingly prevalent.

Experimental design

This experiment uses a between-subjects design with random assignment. Undergraduate participants complete two sessions approximately one week apart. In Session One, participants learn about an unfamiliar academic topic and complete assessments. In Session Two, all participants complete retention assessments.

Design details

Randomization method
Randomization done in office by a computer.
Randomization unit
Individual participant
Clusters
Not applicable (individual-level randomization)
Observations
~200-300 students
Arms
100-150 students in AI-allowed condition, 100-150 students in AI-forbidden condition
Primary outcomes
1. Written assessment quality in Session One (as measured by blind grader evaluations) 2. Post-learning assessment performance in Session One (multiple-choice test) 3. Follow-up written assessment quality in Session Two (as measured by blind grader evaluations) 4. Knowledge retention test performance in Session Two (multiple-choice test) 5. Learning gain (difference between post-learning and baseline assessments)
Secondary outcomes
1. Time allocation during the learning phase 2. AI usage patterns (for treatment group) 3. Computational text metrics (lexical diversity, word count, syntactic complexity) 4. Retention rate (ratio of Session Two to Session One performance)

Cited by

No citation data available for this registration yet.

Keywords