AI Engagement, Belief Formation, and Human Capital Investment: Theory and Experimental Evidence

How to cite

Nguyen, Tuan; The, Nguyen To; Kim, Cuong Pham Thi. 2026. "AI Engagement, Belief Formation, and Human Capital Investment: Theory and Experimental Evidence." AEA RCT Registry. AEARCTR-0018436. https://doi.org/10.1257/rct.18436-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Tuan Nguyen* , Nguyen To The , Cuong Pham Thi Kim

* Primary investigator

Registry ID
AEARCTR-0018436
DOI
https://doi.org/10.1257/rct.18436-1.0
First registered
Last updated
Status
In development
Trial dates
2026-06-01 – 2027-01-31
Countries
Private
JEL codes
I21, D83, C93, D81, J24
Registry record
http://www.socialscienceregistry.org/trials/18436

Abstract

This study examines whether a structured AI-assisted advising session changes students' beliefs about their own academic ability and alters their course enrollment decisions. Students approaching a real course registration decision are randomly assigned to one of three arms: a control group (no session), a productivity arm (AI practice session without feedback), and a feedback arm (AI session with a personalized readiness assessment). The primary outcome is enrollment in a harder course track. Secondary outcomes include belief updating (posterior minus prior pass probability) and end-of-semester grades. The design tests a theoretical model in which AI functions as an endogenous information production technology: students who engage more intensively receive more precise self-assessments, but if the AI system carries an optimistic bias, greater engagement amplifies belief distortion. The experiment is powered to detect a 12-percentage-point shift in hard-course enrollment with 80% power (N ≈ 360).

Intervention

This experiment studies whether a structured AI-assisted advising session, conducted before course registration opens, changes how students assess their own academic readiness and alters the courses they ultimately choose to enrol in. The intervention is built around a single 30-minute session with an AI tool, administered in the week before the registration window opens for the following semester. All students complete a short baseline survey before the session and a post-session survey immediately after. The primary outcome, course enrollment, is obtained from administrative registrar records after the add/drop period closes.
Students are randomly assigned to one of three arms. The control arm receives no structured session. These students proceed through the normal registration process without any intervention from the research team. They complete the baseline and post-session surveys at the same scheduled times as students in the active arms, with the post-session belief question administered after being informed that a session took place for other students and that they are being asked to record their current course planning thinking. This design ensures that any difference between the control arm and the active arms reflects the session itself rather than the act of completing a survey.
The productivity arm attends the 30-minute session and uses an AI tool to work through a set of practice problems drawn from the beginning of the harder course curriculum. The tool is configured to assist with problem-solving, explain underlying concepts, and check the student's reasoning step by step. It is explicitly not configured to comment on the student's ability, predict her likelihood of success in the harder course, or compare her performance to any benchmark or reference group. The session therefore activates a productivity channel, in which AI assistance raises the student's immediate task performance, while holding the belief channel at zero. Students in this arm leave the session having practised relevant material with AI support but without having received any personalised assessment of their academic readiness.
The feedback arm attends an identical 30-minute session using the same AI tool and the same set of practice problems. The session format, the problem difficulty distribution, and the interface are indistinguishable from those in the productivity arm up to the final few minutes of the session. The sole difference is what happens at the end. After completing the problem set, the student receives a personalised readiness assessment generated by the AI on the basis of her observable behaviour during the session. The assessment identifies which categories of problem the student handled confidently and which she struggled with, and it provides a probabilistic estimate of her likely success in the harder course, stated numerically as a percentage chance of earning a passing grade. This assessment is the empirical realisation of the ability signal in the theoretical model. Crucially, its informativeness is endogenous to how the student engaged during the session: a student who attempted harder problems, asked more substantive questions, and iterated on targeted feedback receives a more precise and more reliable assessment than one who attempted only easy questions or interacted with the tool in a surface-level way. This feature distinguishes the feedback treatment from a conventional information intervention that delivers the same message to every student regardless of her behaviour during the session.
Comparing the feedback arm to the control arm identifies the total effect of the structured session on beliefs and enrollment, combining both the productivity and the belief-updating channels. Comparing the feedback arm to the productivity arm isolates the belief channel alone, because the session format, AI tool, problem set, and duration are held constant across these two arms and the only difference is whether the student receives the personalised readiness assessment at the end. This arm comparison is the key identification strategy for the paper's central theoretical claim that AI functions as an endogenous self-knowledge technology rather than a uniform information delivery device.
One important design feature must be stated clearly. Students in the control arm and the productivity arm are not prevented from using AI tools independently during the period between their session and their registration decision. In a contemporary university setting this restriction would be practically unenforceable, and attempting to impose it would introduce ethical complications and differential attrition that would compromise the internal validity of the study. The comparison being made is therefore not between AI access and no AI access, but between a structured, institutionally designed AI advising session and the unstructured status quo in which students make their registration decision using whatever resources they ordinarily draw on. This framing is both honest and policy-relevant, because the practical question facing university administrators is not whether students should use AI at all but whether a structured institutional session adds measurable value over and above unguided use.
The study recruits undergraduate students who are approaching a genuine registration decision in which a harder course is clearly available as an alternative to a standard option. The most natural settings are introductory quantitative sequences, such as introductory statistics, introductory microeconomics, or a foundational mathematics course, where the harder follow-on course is clearly distinguishable from the easier one, where students face genuine uncertainty about whether they are ready for the more demanding option, and where AI-assisted practice on relevant problem types is feasible within a 30-minute session. Students are recruited during the first week of the current semester so that the treatment falls naturally in the period when course selection for the following semester is actively being considered and before any registration decision has been made. Recruitment, random assignment, and all session activities are completed before the registration window opens, ensuring that the intervention precedes rather than follows the outcome it is designed to influence.
The experiment involves no deception. Students in all three arms are informed at recruitment that the study examines how different types of academic support sessions affect course planning decisions, and that their eventual enrollment and grade records will be obtained from the registrar for research purposes. Students in the feedback arm are told before the session begins that they will receive a personalised summary of their performance at the end. Informed consent is obtained from all participants prior to the baseline survey, and the study has received a positive ethics opinion from the Research Ethics Committee of the University of Economics and Business, Vietnam National University, Hanoi (Decision No. 2026-REC-UEB, April 2026).

Experimental design

This is a three-arm individually randomized experiment in which undergraduate students approaching a real course registration decision are randomly assigned to a control condition or one of two active treatment arms before the registration window for the following semester opens. The experiment is conducted within introductory quantitative course sequences where a harder follow-on course is clearly distinguishable from a standard option and where students face genuine uncertainty about their readiness for the more demanding track.
Randomization is stratified by GPA tercile within course section using a pre-registered random seed, which ensures balance on the strongest available predictor of both session engagement and course choice and reduces residual variance in the primary outcome regression. All students complete a five-minute baseline survey before the session begins, covering their self-assessed probability of passing the harder course, risk preferences measured through a five-item Holt and Laury (2002) task and the Dohmen et al. (2011) single-item scale, prior experience with AI tools, self-assessed proficiency in the relevant subject area, and grade-following behaviour. All students complete a five-minute post-session survey immediately after the session, covering the same pass-probability question worded identically to the baseline version, a self-report of session engagement, and, for feedback-arm students only, two manipulation check questions about whether they recall the readiness estimate they received and how accurate they found it. Cumulative GPA and credits completed are obtained from administrative records without self-report.
The primary behavioral outcome is the course the student actually enrolls in, obtained from registrar records after the add/drop period closes for the following semester. The secondary welfare outcome is the grade the student earns in that course, obtained from registrar records at the end of the following semester. No outcome data are collected by the research team; both outcomes arrive from administrative sources and require no further contact with participants after the post-session survey.
The planned sample is 360 students across three equally sized arms of 120 students each. The power calculation targets the comparison between the feedback arm and control on the hard-course enrollment indicator, with a baseline enrollment rate of 35 percent, a minimum detectable effect of 12 percentage points, a two-sided significance level of 0.025 after Bonferroni correction for two arm comparisons, and 80 percent power. If single-cohort recruitment falls short of the target, the design pools across two consecutive cohorts while preserving stratified randomization within each cohort and including cohort fixed effects in all regression specifications. The full pre-analysis plan, specifying all primary hypotheses, estimating equations, covariate lists, and the falsification restriction that the first-stage coefficient on the feedback arm indicator must strictly exceed that on the productivity arm indicator, is registered before any data collection begins.

Design details

Randomization method
Randomization is conducted by computer in the research office before recruitment begins, using a pre-registered random seed documented in the pre-analysis plan. Stratification proceeds by first dividing students within each course section into three equally sized GPA terciles based on cumulative GPA obtained from administrative records, and then independently randomizing within each tercile-by-section cell using a fixed random seed. This procedure ensures that the three arms are balanced on GPA, which is the strongest available predictor of both session engagement and hard-course enrollment, and that the stratification structure is fully transparent and replicable from the pre-registered seed. Assignment ratios within each cell are fixed at 1:1:1 across the three arms. The assignment list is generated and sealed before any student contact occurs and is not accessible to session supervisors during the recruitment or session period.
Randomization unit
Individual student. The design is not clustered. All students within the same course section are eligible for all three arms, and section membership is accounted for by including section fixed effects in all regression specifications. There is no group-level randomization and no situation in which students within the same section are all assigned to the same arm. The absence of clustering reflects the practical reality that the 30-minute sessions are conducted individually or in small groups that do not interact, and the treatment, whether or not to receive the readiness assessment, is delivered at the individual level with no spillover pathway to other students in the same section.
Clusters
360 individual students. Because the unit of randomization is the individual and the design is not clustered, the number of clusters equals the number of observations.
Observations
360 students across three arms of 120 students each. If single-cohort recruitment falls short of 360 due to lower-than-expected enrollment in the target course sequences, the design pools across two consecutive cohorts up to a maximum of 720 students, with cohort fixed effects included in all specifications and the stratified randomization procedure applied independently within each cohort.
Arms
120 students assigned to the control arm, 120 students assigned to the productivity arm, and 120 students assigned to the feedback arm, for a total of 360 students.
Primary outcomes
Hard-course enrollment indicator, posterior belief update, and end-of-semester grade in enrolled course.
Secondary outcomes
Session engagement intensity, readiness assessment accuracy for feedback-arm students, and heterogeneity of belief revision by prior uncertainty level.

Cited by

No citation data available for this registration yet.

Keywords