Boundaries of Generative AI Capabilities in Multi-tasking Environments
How to cite
Zhu, Zhenyu; Wu, Yanhui. 2026. "Boundaries of Generative AI Capabilities in Multi-tasking Environments." AEA RCT Registry. AEARCTR-0019108. https://doi.org/10.1257/rct.19108-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Recent advancements in Generative AI (GenAI) have the potential to significantly enhance productivity and reshape workflows. A critical debate in exposed industries is whether GenAI should primarily be deployed as a labor-augmenting copilot, or as an autonomous agent that directly substitutes for human workers. In practice, jobs are typically designed as task bundles to balance specialization gains against coordination costs. Therefore, evaluating GenAI as an autonomous worker requires assessing not only its performance on isolated tasks, but also its ability to handle a multi-task bundle. To study this topic, we collaborate with a leading customer service company. Customer service is a typical multi-task job, comprising a bundle of tasks ranging from product consultation to personalized recommendation. These tasks differ fundamentally in their nature: some are standardized and require strict factual accuracy, while others are creative and require strategic persuasion to close the sale. To systemically evaluate GenAI’s performance in isolated tasks and task bundling relative to human workers, we will run three randomized controlled experiments (RCTs) that vary the degree of task bundling while holding other settings constant. Through these investigations, this study aims to inform the optimal design of human-AI workflows, providing micro-foundations for understanding the boundaries of autonomous agents versus augmenting copilots in complex work environments.
Intervention
We will run three experiments varying in the task composition. In each experiment, we generate a set of scenario cards, each specifying the consumer demands embedded in a complete inquiry. The construction and assignment protocols are detailed in the next section. These cards are assigned to “consumers” (played by hired students) and service agents (GenAI or humans) following a fixed algorithm. Throughout the study, the GenAI agent serves as the treatment group, while human workers serve as the control benchmark.
Experimental design
Customer service agents are divided into two groups: human agents and the GenAI agent. Each student is assigned 30 scenario cards, allocated evenly across treatment arms: 15 cards are designated for interactions with human agents, and 15 for the GenAI agent. We employed the greedy assignment algorithm to ensure that
• student characteristics are balanced across tasks;
• student characteristics are balanced between human and GenAI agents;
• task distribution is balanced between human and GenAI agents.
All human agents are required to participate in the experiment. Both human and GenAI agents serve students on a strictly one-on-one basis (i.e., no concurrent sessions). We maintain a constant active roster of three human agents and one GenAI agent account throughout the experiment. The daily sequence of interactions between students and agents (human vs. GenAI) is randomized to eliminate potential order effects. Agent identities are concealed to ensure a blinded design, meaning students cannot distinguish whether they are conversing with a human or the GenAI agent.
Design details
Cited by
No citation data available for this registration yet.