Conditions of AI Engagement in Learning: An Online Experiment
How to cite
Kinney, Adrienne; Abel, Martin; Workman, Anya; Yaspik, Vianney Gomezgil. 2026. "Conditions of AI Engagement in Learning: An Online Experiment." AEA RCT Registry. AEARCTR-0019322. https://doi.org/10.1257/rct.19322-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This study examines how the incentives participants anticipate, and the time available to them, shape the way they engage with AI assistance while studying, and how those engagement patterns relate to later unaided performance. Adults recruited from Prolific complete a two-phase task. In the first phase, participants answer multiple-choice questions drawn from several academic subjects areas, with optional assistance available on request for each question. In the second phase, they answer related questions with no assistance available and with accuracy incentivized by a bonus payment. Participants are randomly assigned with equal probability to one of five arms, which differ the task conditions surrounding the first phase. Outcomes include the depth and patterns of assistance use in the first phase, time spent per item, and accuracy in the unassisted phase. The design is intended to distinguish assistance use that supports independent performance from assistance use that substitutes for it.
Intervention
Participants complete a computer-administered two-phase task. In the study phase, they answer four multiple-choice questions, one from each of four academic domains, presented one per page in individually randomized order. In four of the five arms, an optional on-demand menu of AI-generated hints is available beside each question; participants choose freely whether and how much to consult it, and no correctness feedback is given. In the fifth arm no hints are available. In the test phase, all participants answer a further set of questions on related material with no assistance available.
Participants are randomly assigned with equal probability to one of five arms. The arms differ in the conditions under which the study phase is completed. Those conditions are specified in the non-public intervention field and in the pre-analysis plan, and will be released when the trial is complete.
All hint content is pre-generated by the research team using an AI language model, reviewed for accuracy, and embedded as fixed text in the survey instrument; no participant data are transmitted to any external AI system at any point. Correct answers earn a per-item bonus in both phases, in every arm.
Experimental design
Between-subjects randomized experiment conducted online with approximately 2,000 participants recruited through Prolific, targeting approximately 400 per arm. Eligibility is restricted to United States residents aged 18–25 with at least some college education, prescreened by Prolific and re-verified within the survey instrument.
Participants first complete a baseline block comprising an instructed-response attention check and self-rated confidence in each of the four task domains. They are then randomly assigned with equal probability to one of five arms using the survey platform's randomizer. Randomization is at the individual level; assignment is stored as an embedded field and governs all subsequent survey branching.
One arm serves as a pure control in which no assistance is available. The remaining four arms, in all of which assistance is available, form a 2×2 between-subjects factorial over two manipulated task conditions. The identity of those two conditions is specified in the non-public design field and in the pre-analysis plan, and will be made public when the trial is complete. The control arm identifies the overall effect of assistance availability relative to working unaided; the factorial identifies the main effect of each manipulation within assistance-available arms, and their interaction.
Following the study phase, participants complete brief post-task ratings and report a domain-by-domain expectation of their own performance on the forthcoming test, then complete the unassisted test phase, followed by a validity check and a demographic block, both administered after all outcomes are measured.
Analysis is intent-to-treat. Person-level outcomes are estimated by OLS on arm indicators with the control condition as the reference category, using heteroskedasticity-robust standard errors, and reported both with and without a vector of baseline covariates. Question-level process outcomes are estimated with question fixed effects and standard errors clustered by participant. The factorial contrasts are estimated on assistance-available arms only, in both a saturated specification including the interaction and a restricted specification omitting it, always reported together. Pre-specified exclusions cover the attention check, the eligibility criteria, incomplete responses, duplicate submissions from a single platform identifier, and a post-outcome validity check; results including participants failing the validity check are reported as a robustness check.
Design details
Cited by
No citation data available for this registration yet.