AI Assistance and Student Problem Solving: The Role of Timing and Accuracy

How to cite

Mumcu, Ipek; Kubilay, Elif. 2026. "AI Assistance and Student Problem Solving: The Role of Timing and Accuracy." AEA RCT Registry. AEARCTR-0018411. https://doi.org/10.1257/rct.18411-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Ipek Mumcu* , Elif Kubilay

* Primary investigator

Registry ID
AEARCTR-0018411
DOI
https://doi.org/10.1257/rct.18411-1.0
First registered
Last updated
Status
In development
Trial dates
2026-04-20 – 2026-06-26
Countries
Private
JEL codes
C93, D83, I20
Registry record
http://www.socialscienceregistry.org/trials/18411

Abstract

This study explores how access to AI-generated solutions influences problem-solving behavior and performance in 16–18-year-old students. Participants are randomly assigned to different conditions based on when they receive AI help and whether the AI solution is correct or incorrect, plus a no-AI control group. They complete an incentivized task with AI access and a second task without it to measure effects on independent performance.

Intervention

The intervention consists of providing participants with access to an AI-generated solution while completing a simple problem-solving task. The intervention varies along two dimensions: the timing of access to the AI-generated solution and the accuracy of the solution provided. Participants may receive access either immediately or delayed, and the AI-generated solution may be either correct or incorrect. Participants in the pure control group receive no AI assistance while solving the problem.

Experimental design

The experimental design follows a 2×2 between-subjects framework with an additional pure control condition. Participants are randomly assigned to conditions that vary along two dimensions: the timing of AI access (immediate versus delayed) and the accuracy of the AI-provided solution (correct versus incorrect), or to a control condition with no AI access. Participants are randomly assigned at the individual level with equal probability across all five conditions. Participants complete a simple, incentivized problem-solving task. In the immediate access condition, participants can request an AI-generated solution at any point from the start of the task. In the delayed access condition, the AI-generated solution is not immediately accessible and becomes available during the course of the task, with the point of access standardized across all participants in this condition. Within each timing condition, participants are randomly assigned to receive either a correct or an incorrect AI-generated solution upon request. Participants in the control condition are not given any access to AI assistance and are required to solve the task independently. This results in five groups defined by the combination of access timing, solution accuracy, and the absence of AI access.

Group 1: Pure control (no AI access)
Group 2: Immediate access – correct solution
Group 3: Immediate access – incorrect solution
Group 4: Delayed access – correct solution
Group 5: Delayed access – incorrect solution

All participants are first presented with a problem and instructed to solve it within a fixed time limit. After completing the first task, all participants proceed to a second problem. In this second stage, all participants are treated identically regardless of their initial assignment. They are given a fixed time limit to solve the problem and are not provided with any AI assistance. Both stages are incentivized.

Design details

Randomization method
Randomization at the individual level via lottery. We implement stratified randomization by session to ensure balanced allocation of participants across treatment arms within each classroom.
Randomization unit
individual level
Clusters
Around 2500 high school students
Observations
Around 2500 high school students
Arms
Around 500 high school students in each of the 5 treatment arms.
Primary outcomes
The primary outcomes are binary indicators of whether the participant submitted a correct answer in the first stage and in the second stage.
Secondary outcomes
From the first stage of the task, the following are used as secondary outcomes: ⁃ AI usage behavior ⁃ Task completion time ⁃ An indicator for whether the participant entered a response prior to viewing the AI-generated solution ⁃ An indicator for whether the participant revised their answer after viewing the AI-generated solution ⁃ The time elapsed between the appearance of the AI-generated solution and the first subsequent edit to the answer ⁃ In cases where the participant’s answer differed from the AI-generated solution, an indicator for whether the participant subsequently revised their answer In addition, participants’ reported confidence in their submitted answers in both stages is included as a secondary outcome.

Cited by

No citation data available for this registration yet.

Keywords