RCT for AI Agent: Cybernetics and Human Capital Talent Growth
How to cite
Han, Tianyang; Luo, Ye; Wu, Yuxiao. 2025. "RCT for AI Agent: Cybernetics and Human Capital Talent Growth." AEA RCT Registry. AEARCTR-0015139. https://doi.org/10.1257/rct.15139-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
The rapid growth of Large Language Models (LLMs) is gradually changing the way human learn and work. Nowadays, with more developed AI Agent, many tedious workflows can be largely simplified into simple human language instruction. With strong AI Agents that both master domain specific knowledge and provide automatic implementation, users can directly deliver a complete outcome for a given task without having to spend time and effort in mastering relevant knowledge.
While automation could directly boost productivity towards time-consuming work, its performance towards creative work stays not that satisfactory. AI researchers have been dreaming of creating robots to complete all work such that human could have time to think, to write, to compose and to create; but till now, the indirect influence of AI Agent application towards human capital knowledge level and creativity level has not receive sufficient attention or discussion:
1. With the help of AI Agent, users are no longer forced to immediately master relevant knowledge to complete time-consuming tasks on hand. Will users’ long-term knowledge level therefore be negatively affected? Will there be any heterogeneity within the effect (such as users’ background relevance and study ability)?
2. With the help of AI Agent, users can greatly save time from the time-consuming tasks. How will users’ performance towards creative work be affected? How will AI Agent’s effect on users’ knowledge level intervene in the procedure? Is the application of AI Agent eventually a good thing for human capital creativity?
We generate the following hypotheses:
Hypothesis 1: After applying AI Agent, users tend to have long-term underperformance on relevant domain knowledge, with the results stronger for more specialized/professional AI Agent design;
Hypothesis 2: After applying AI Agent, users will perform better on creative work, ceteris paribus;
Hypothesis 3: Knowledge level will endogenously affect creativity level: after applying AI Agent, users’ increase in creative work performance will be partially offset by the decrease of domain knowledge level, with the results stronger for more specialized AI Agent design;
Hypothesis 4: All of above effects will be heterogeneous based on users’ knowledge background and learning capability.
Intervention
In this RCT, the intervention is designed to control students’ access to our well-prepared AI Agent tool in completing course assignments. Treatment group students can and are encouraged to use AI Agent to solve the knowledge-based task in their corresponding assignment, while control group students need to solve all questions in the assignment by themselves. Two types of AI Agents are randomly assigned to the treatment group each time (i.e., two subgroups in the entire treatment group):
• General AI Agent can complete an entire workflow but is not equipped with completely developed domain-knowledge tools.
• Specialized AI Agent, upgraded from general AI Agent with well-equipped domain-knowledge tools
Experimental design
Step 0 (Course Setup): Set three assignments throughout this course. The first assignment contains knowledge-driven questions. The second and third assignments each contain two questions, one about more theory knowledge work (for example, Fama-Macbeth regression, Machine Learning theory and application, etc.) and the other one containing more creative work. Also there will be a written final exam after all lectures and assignments. The 3-hour exam includes questions containing the theory knowledge from the three assignments. The final exam must be close-book.
• HW1: Factor model & Fama-Macbeth method (knowledge driven)
• HW2: Investment portfolio analysis (knowledge driven); Factor model & Fama-Macbeth method exploration (knowledge based on HW1; creative tasks)
• HW3: Machine learning practice (knowledge driven); Machine learning methodology explo- ration & creation (knowledge based on HW3; creative tasks)
Step 1: Prof. Ye LUO at the first lecture announces the existence of AI Agent availability towards both subclasses, introduce AI Agent briefly and announce that students will be taught about AI Agent and given accessible accounts in the class. After the first lecture, tutors will
distribute questionnaires to collect information from students. The information will serve as the pre-determined control variables in our future analysis. Questions will include (but may not limit to):
• Students’ previous experience towards LLM & AI Agent;
• Programming experience;
• Previous education/knowledge background;
• (Self-evaluation) Personal interest on research & in-depth thinking.
Step 2: For Assignment 1, no AI Agent will be provided and all the questions are knowledge-driven.We shall collect students’ performance in Assignment 1.
Step 3: After Assignment 1, conditional on students’ performance in Assignment 1, as well as students’ background match, we randomly split all the students into two groups. The first group (Group 1) will be trained before the announcement of Assignment 2. At the same time, each of them will get their AI Agent personal accounts. Two tutorial sessions will be held for Group 1. One sub-session will be taught standard AI Agent, while the other will be taught the specialized AI Agent. Students will not be told that two sub-sessions are provided different types of AI Agents. The second group (Group 2) will not get their AI Agent account or receive AI Agent training for
Assignment 2.
Step 4: For Assignment 2, we shall collect students’ performance for both groups. The AI Agent account will be suspended for Group 1 once the DDL of Assignment 2 approaches. A short survey will be distributed to all students, with self-evaluation questions including (but may not limit to):
• Time consumption on both questions;
• (Only for Group 1) The contribution of AI Agent towards two questions;
• (Only for Group 1) The time saving of AI Agent towards two questions;
• (Only for Group 2) Two placebo questions.
Step 5: After Assignment 2, Group 2 will be trained before the announcement of Assignment 3. At the same time, each of them will get their AI Agent personal accounts until the DDL of Assignment 3. Two tutorial sessions will be held for Group 2. One sub-session will be taught
standard AI Agent, while the other will be taught the specialized AI Agent. Students will not be told that two sub-sessions are provided different types of AI Agents. Group 1 at this time will not get their AI Agent account or receive AI Agent training for Assignment 3.
Step 6: For Assignment 3, we shall collect students’ performance for both groups. The AI Agent account will be suspended for Group 2 once the DDL of Assignment 3 approaches. A short survey will be distributed to all students, with self-evaluation questions including (but may not limit to):
• Time consumption on both questions;
• (Only for Group 2) The contribution of AI Agent towards two questions;
• (Only for Group 2) The time saving of AI Agent towards two questions;
• (Only for Group 1) Two placebo questions.
Step 7: The final exam will be close-book and will cover all knowledge contents covered in the three assignments. We shall record all students’ performance for further analysis. After final exam, tutors will distribute questionnaires to collect self-evaluation information from students. Questions will include (but may not limit to):
• The effect of AI Agent application towards knowledge learning.
Design details
Cited by
No citation data available for this registration yet.