From Autopilot to Co-Pilot: Guiding Effective AI Use in Higher Education

How to cite

Shaikh, Hammad; Keskar, Ajinkya; Tonguc, Ozlem; Tatro, Case. 2026. "From Autopilot to Co-Pilot: Guiding Effective AI Use in Higher Education." AEA RCT Registry. AEARCTR-0017652. https://doi.org/10.1257/rct.17652-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Hammad Shaikh* , Ajinkya Keskar , Ozlem Tonguc , Case Tatro

* Primary investigator

Registry ID
AEARCTR-0017652
DOI
https://doi.org/10.1257/rct.17652-1.0
First registered
Last updated
Status
On going
Trial dates
2026-04-06 – 2026-12-18
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/17652

Abstract

The recent introduction of generative AI (e.g., ChatGPT) has sparked contentious debates among educators over whether to adopt or ban it. This study aims to provide empirical evidence on whether training and incentivising students to use generative AI as a complementary productivity-enhancing technology (rather than a substitute) for their own effort can increase educational attainment. In a lab experiment, students learning about economic literacy are randomized into four groups: (1) No AI, (2) AI, (3) AI and training, and (4) AI, training, and incentives. The training treatment teaches students evidence-based prompt engineering techniques and encourages students to use AI in a complementary way. Additionally, some students are given a bonus payment for writing high-quality prompts as instructed in the training. Our key outcomes include whether students integrate prompt engineering techniques, write prompts that complement their own effort, and their performance on a closed-book exam. The results of the study will provide evidence on whether students can be trained to use AI more effectively and what effects this has on their learning.

Intervention

Students are randomized into one of four groups. The control group learns traditionally without AI. Then there are three treatment groups (1) AI Access, (2) AI Access + Prompt Engineering Training, and (3) AI Access + Prompt Engineering Training + Incentives for Good Prompts.

The AI training module is a 10-minutes video which begins by highlighting the importance of critical thinking (providing research evidence showing returns to critical thinking over memorized knowledge) and encouraging students to use AI as a complement, rather than a substitute. Then the training transitions into prompt engineering, where they are taught 3 useful and evidence-based prompt engineering techniques and shown examples. The three techniques include (i) providing a role and context, (ii) breaking down problems into smaller steps, (3) giving examples and generating practice problems. Finally, in the final segment of the training, students are asked to improve low-quality prompts and integrate the taught techniques.

To provide incentives to students to utilize and focus on the AI training, students in the final treatment group are given a bonus payment based on the quality of prompts they write. If they integrate some of the prompt engineering techniques, they are awarded with a bonus payment.

Experimental design

The experimental design has the following four groups:

(1) No AI (traditional learning)
(2) AI access (free chatGPT)
(3) AI access + AI effective use training
(4) AI access + AI effective use training + bonus payment for prompts that implement practices taught in the training

Design details

Randomization method
We will do a stratified randomization. We stratify respondents on a 2 x 3 grid using (i) baseline academic preparation and (ii) baseline frequency of AI use. Academic preparation is a binary index built from current GPA, SAT score,  prior high-school economics, and self-rated economics proficiency. On the other hand, AI use is grouped into none, occasional/unknown, and regular/extensive. We have 6 stratas in total.
Randomization unit
The randomization unit is the student.
Clusters
No clusters, but we may have up to 40 lab sessions.
Observations
1000 students is the ideal target. If we have recruiting difficulties, we will aim for around 500.
Arms
For example, N = 1000: 400 in the control group (no AI) and 200 in all three treatment groups, around 20 students per lab session. Smaller samples will keep the same distribution.
Primary outcomes
We have both survey and administrative outcomes, enabling us to have both self-reported and actually observed AI usage and prompting strategies. Survey outcomes: - Assigned a tutor/TA role to the AI (yes/no) - Utilizing prompt engineering techniques (index) - Frequency of copy-pasting problems without trying them (5-point Likert scale) Administrative outcomes (some outcomes based on students' prompt logs): - Any AI used - Percent of prompts that do not copy-paste the question - Percent of prompts that utilize at least one prompt engineering technique (role, context, steps, or example) - Score on the closed-book exam

Cited by

No citation data available for this registration yet.

Keywords