Boundaries of Generative AI Capabilities in Multi-tasking Environments

How to cite

Zhu, Zhenyu; Wu, Yanhui. 2026. "Boundaries of Generative AI Capabilities in Multi-tasking Environments." AEA RCT Registry. AEARCTR-0019108. https://doi.org/10.1257/rct.19108-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Zhenyu Zhu* , Yanhui Wu

* Primary investigator

Registry ID
AEARCTR-0019108
DOI
https://doi.org/10.1257/rct.19108-1.0
First registered
Last updated
Status
On going
Trial dates
2026-03-28 – 2026-10-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/19108

Abstract

Recent advancements in Generative AI (GenAI) have the potential to significantly enhance productivity and reshape workflows. A critical debate in exposed industries is whether GenAI should primarily be deployed as a labor-augmenting copilot, or as an autonomous agent that directly substitutes for human workers. In practice, jobs are typically designed as task bundles to balance specialization gains against coordination costs. Therefore, evaluating GenAI as an autonomous worker requires assessing not only its performance on isolated tasks, but also its ability to handle a multi-task bundle. To study this topic, we collaborate with a leading customer service company. Customer service is a typical multi-task job, comprising a bundle of tasks ranging from product consultation to personalized recommendation. These tasks differ fundamentally in their nature: some are standardized and require strict factual accuracy, while others are creative and require strategic persuasion to close the sale. To systemically evaluate GenAI’s performance in isolated tasks and task bundling relative to human workers, we will run three randomized controlled experiments (RCTs) that vary the degree of task bundling while holding other settings constant. Through these investigations, this study aims to inform the optimal design of human-AI workflows, providing micro-foundations for understanding the boundaries of autonomous agents versus augmenting copilots in complex work environments.

Intervention

We will run three experiments varying in the task composition. In each experiment, we generate a set of scenario cards, each specifying the consumer demands embedded in a complete inquiry. The construction and assignment protocols are detailed in the next section. These cards are assigned to “consumers” (played by hired students) and service agents (GenAI or humans) following a fixed algorithm. Throughout the study, the GenAI agent serves as the treatment group, while human workers serve as the control benchmark.

Experimental design

Customer service agents are divided into two groups: human agents and the GenAI agent. Each student is assigned 30 scenario cards, allocated evenly across treatment arms: 15 cards are designated for interactions with human agents, and 15 for the GenAI agent. We employed the greedy assignment algorithm to ensure that
• student characteristics are balanced across tasks;
• student characteristics are balanced between human and GenAI agents;
• task distribution is balanced between human and GenAI agents.

All human agents are required to participate in the experiment. Both human and GenAI agents serve students on a strictly one-on-one basis (i.e., no concurrent sessions). We maintain a constant active roster of three human agents and one GenAI agent account throughout the experiment. The daily sequence of interactions between students and agents (human vs. GenAI) is randomized to eliminate potential order effects. Agent identities are concealed to ensure a blinded design, meaning students cannot distinguish whether they are conversing with a human or the GenAI agent.

Design details

Randomization method
Customer service agents are divided into two groups: human agents and the GenAI agent. Each student is assigned 30 scenario cards, allocated evenly across treatment arms: 15 cards are designated for interactions with human agents, and 15 for the GenAI agent. We employed the greedy assignment algorithm to ensure that • student characteristics are balanced across tasks; • student characteristics are balanced between human and GenAI agents; • task distribution is balanced between human and GenAI agents.
Randomization unit
Randomization is conducted at the individual level.
Clusters
Our experiment does not need to cluster the treatment.
Observations
1228
Arms
574 inquiries are assigned to human agents, and 654 inquiries are assigned to AI agents.
Primary outcomes
The key outcome is the agent performance in customer service.

Cited by

No citation data available for this registration yet.

Keywords