Justification for AI Use - A Field Experiment with LLM Agents

How to cite

Grabe, Leonhard; Lübke, Simon. 2025. "Justification for AI Use - A Field Experiment with LLM Agents." AEA RCT Registry. AEARCTR-0017259. https://doi.org/10.1257/rct.17259-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Leonhard Grabe* , Simon Lübke

* Primary investigator

Registry ID
AEARCTR-0017259
DOI
https://doi.org/10.1257/rct.17259-1.0
First registered
Last updated
Status
In development
Trial dates
2025-11-23 – 2025-12-31
Countries
Private
JEL codes
M11, M21, O33
Registry record
http://www.socialscienceregistry.org/trials/17259

Abstract

In a field experiment embedded in regular outbound customer service operations, we examine whether a small framing intervention improves customer engagement during fully automated calls conducted by an AI voice system. In these calls, the AI system handles the complete conversation flow and may transfer customers to human agents only when necessary. We compare call outcomes between a control group that receives the standard neutral introductory sentence and a treatment group that receives an enhanced introduction explaining why the AI system is calling and emphasizing the intended benefit for the customer, namely quicker processing of their request. We evaluate whether this additional explanatory sentence increases the likelihood that customers stay in the call, reduces immediate hang-ups, and affects subsequent transfers to human agents. The experiment relies on call-level random assignment and uses only operational data already collected for quality assurance. The aim is to assess whether transparent, benefit-oriented communication about the AI system’s role can measurably improve customer interaction quality in fully automated service calls.

Intervention

We evaluate the effect of using justifications for AI use on the acceptance of AI chatbots. We modify the introduction of an LLM-based chatbot, whose task it is to arrange appointments on behalf of a large technology company via phone call. In the control condition, the agent introduces itself by name and states the purpose of the call. In the treatment condition, we include an additional sentence to justify the delegation of this task to the AI, such as earlier appointments for the consumer. The key outcome variable is the likelihood of succesfully scheduling an appointment with the chatbot.

Experimental design

We evaluate the effect of providing justifications for AI use on the acceptance of AI chatbots. The study is conducted using an LLM-based chatbot, whose task it is to arrange appointments on behalf of a large technology company via phone call. We modify whether or not the bot provides and explanation for why the bot was tasked with scheduling the appointments rather than a human customer service agent. We measure the acceptance of the bot using the share of successfully booked appointments and transferred calls, as well as the average duration and sentiment of the call - if this data is available.

Design details

Randomization method
Customers are randomly allocated to one of two AI agents by the firm using a digital coinflip.
Randomization unit
Randomization was implemented on the individual level and day-level. That is, all customers that are scheduled to be contacted on a given day are allocated to one group.
Clusters
8000 customers (4000 per treatment group)
Observations
8000 customers (4000 per treatment group)
Arms
4000 customers
Primary outcomes
The key outcome is a binary indicator whether the agent was successful in scheduling an appointment with the customer (=1) or not (=0).
Secondary outcomes
Secondary outcomes include whether the call was transferred to a human agent, the total duration of the call, and the share of calls that are terminated without a successful outcome after less than one minute (if the data is available).

Cited by

No citation data available for this registration yet.

Keywords