Agentic AI and Structured vs. Self-guided learning

How to cite

Bapna, Sofia; Bapna, Ravi; Yang, Mochen; Umyarov, Akhmed; Rama, Vandith Pamuru Subramanya. 2025. "Agentic AI and Structured vs. Self-guided learning." AEA RCT Registry. AEARCTR-0016373. https://doi.org/10.1257/rct.16373-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Sofia Bapna* , Ravi Bapna , Mochen Yang , Akhmed Umyarov , Vandith Pamuru Subramanya Rama

* Primary investigator

Registry ID
AEARCTR-0016373
DOI
https://doi.org/10.1257/rct.16373-1.0
First registered
Last updated
Status
In development
Trial dates
2025-07-24 – 2026-08-01
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/16373

Abstract

Agentic-AI tools are beginning to be integrated into classrooms to support student learning and prepare them for a workforce where such tools are increasingly common. An agentic-AI tool is deployed in a core analytics course at a top-30 global MBA program; the tool serves as a virtual data scientist and is used by students to solve case assignments. We test which type of guidance is most effective for students using the tool. Students are randomly assigned to one of two conditions: a Structured Learning condition and a Self-guided Learning condition. In each condition, students receive either exact prompts to use for the case or a description of the tool’s capabilities as they pertain to the case. In both conditions, the guidance is intended only to help students get started on solving the assigned cases; neither provides a complete solution. We examine (1) which type of agentic-AI usage guidance improves learning outcomes (captured by performance on a proctored, in-person, closed-book, pen-and-paper final exam), (2) whether usage differs across students based on the type of guidance, and (3) whether the type of guidance has heterogeneous effects on learning outcomes and usage patterns by gender, competence (e.g., prior academic performance), and work experience (e.g., analytics experience).

Intervention

Students are randomly assigned to one of two conditions: Structured Learning or Self-guided Learning.

There are two individual case assignments in the course, posted one week apart. For each assignment, students receive an email from the instructor corresponding to their assigned experimental condition. Once assigned to a condition, students remain in that condition for both assignments. Emails are sent as soon as the assignment is posted on the LMS. Students have four days to complete each assignment (e.g., if the assignment is posted on Thursday (or Friday), the solution is due on Monday (or Tuesday), respectively).

There are multiple sections of the course; some sections post the assignment on Thursday and others on Friday.

The subject line of the emails is: “Instructions for your assignment due on <Month> <Day>.” The email templates are shown below.

Note: ‘Opening Message’ and ‘Instructor Name’ are placeholders that will be kept constant across conditions for each case. Similarly, <Month> <Day> are placeholders indicating the assignment due date.

*Structured Learning - Case 1*
Dear students,
[Opening Message 1] The “Cancer Detection” case on binary classification is now available on the DAB platform. We encourage you to start chatting with the AI agent to learn about this case. To help you get started, consider using the following prompts:
1. Please describe the case.
2. Please provide a correlation heat map of the input variables.
3. Please build a logistic regression model to detect cancer and show the detailed steps.
4. Please show me the first 10 rows in the test set with actual outcome, predicted probability, and predicted class label. Format this as a table.
5. Please test the kNN model for this prediction task. What would be a good value of k?
Best regards,
[Instructor Name]

*Self-guided Learning Case 1*
Dear students,
[Opening Message 1] The “Cancer Detection” case on binary classification is now available on the DAB platform. We encourage you to start interacting with the AI agent to learn about this case. To help you get started, consider exploring the following capabilities of the AI agent:
1. The AI agent can provide a description of the case.
2. The AI agent can create various visualizations of the data.
3. The AI agent can perform manipulation and processing on the data.
4. The AI agent can build and evaluate numeric prediction models, such as linear regression and kNN, on the specific outcome variable you choose.
5. The AI agent can perform systematic model tuning to find a good performing model.
Best regards,
[Instructor Name]

*Structured Learning - Case 2*
Dear students,
[Opening Message 2] The “Catalog Company” case on numeric prediction is now available on the DAB platform. We encourage you to start chatting with the AI agent to learn about this case. To help you get started, consider using the following prompts:
1. Please describe the case.
2. Can you visualize the distribution of the “Spending” variable in a histogram?
3. Build a linear regression model to predict spending. Show the detailed process and report the RMSE and MAPE metrics.
4. Please redo the model by not using the purchase variable and report the full model as well as RMSE.
5. Please build a KNN model to predict spending. Choose a good value k and report RMSE.
Best regards,
[Instructor Name]

*Self-guided Learning Case 2*
Dear students,
[Opening Message 2] The “Catalog Company” case on numeric prediction is now available on the DAB platform. We encourage you to start interacting with the AI agent to learn about this case. To help you get started, consider exploring the following capabilities of the AI agent:
1. The AI agent can provide a description of the case.
2. The AI agent can create various visualizations of the data.
3. The AI agent can perform manipulation and processing on the data.
4. The AI agent can build and evaluate various numeric prediction models, such as linear regression and kNN.
5. The AI agent can perform systematic model tuning to find a good performing model.
Best regards,
[Instructor Name]

Experimental design

An agentic-AI tool is deployed in a core analytics course in an MBA program, serving as a virtual data scientist to help students solve case assignments. We test the effect of different types of guidance—Structured Learning or Self-guided Learning—on how students use the tool to identify solutions to the assignments. Students are randomly assigned to one of the two conditions and receive case assignment emails from the instructor aligned with their assigned condition. We measure how this guidance influences both students’ usage of the tool and their learning outcome. Additionally, we will examine heterogeneity by gender, competence (e.g., previous academic performance), work experience (e.g analytics experience).

Design details

Randomization method
Simple randomization using a script (with a seed, so the randomization can be replicated).
Randomization unit
Individual (student). The students are distributed across multiple sections of the course. Randomization to the two conditions will occur within each section. Across all students (combining students across sections) we will check for balance on: gender, sector of experience, years of experience, and undergrad degree type (assuming we receive this data).
Clusters
N/A
Observations
All students registered in the course. About 200.
Arms
Each treatment arm will have an equal (or nearly equal) number of students.
Primary outcomes
Final exam score, which is a proxy for learning.
Secondary outcomes
Number of prompts

Cited by

No citation data available for this registration yet.

Keywords