Justification for AI Use - A Field Experiment with LLM Agents
How to cite
Grabe, Leonhard; Lübke, Simon. 2025. "Justification for AI Use - A Field Experiment with LLM Agents." AEA RCT Registry. AEARCTR-0017259. https://doi.org/10.1257/rct.17259-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
In a field experiment embedded in regular outbound customer service operations, we examine whether a small framing intervention improves customer engagement during fully automated calls conducted by an AI voice system. In these calls, the AI system handles the complete conversation flow and may transfer customers to human agents only when necessary. We compare call outcomes between a control group that receives the standard neutral introductory sentence and a treatment group that receives an enhanced introduction explaining why the AI system is calling and emphasizing the intended benefit for the customer, namely quicker processing of their request. We evaluate whether this additional explanatory sentence increases the likelihood that customers stay in the call, reduces immediate hang-ups, and affects subsequent transfers to human agents. The experiment relies on call-level random assignment and uses only operational data already collected for quality assurance. The aim is to assess whether transparent, benefit-oriented communication about the AI system’s role can measurably improve customer interaction quality in fully automated service calls.
Intervention
We evaluate the effect of using justifications for AI use on the acceptance of AI chatbots. We modify the introduction of an LLM-based chatbot, whose task it is to arrange appointments on behalf of a large technology company via phone call. In the control condition, the agent introduces itself by name and states the purpose of the call. In the treatment condition, we include an additional sentence to justify the delegation of this task to the AI, such as earlier appointments for the consumer. The key outcome variable is the likelihood of succesfully scheduling an appointment with the chatbot.
Experimental design
We evaluate the effect of providing justifications for AI use on the acceptance of AI chatbots. The study is conducted using an LLM-based chatbot, whose task it is to arrange appointments on behalf of a large technology company via phone call. We modify whether or not the bot provides and explanation for why the bot was tasked with scheduling the appointments rather than a human customer service agent. We measure the acceptance of the bot using the share of successfully booked appointments and transferred calls, as well as the average duration and sentiment of the call - if this data is available.
Design details
Cited by
No citation data available for this registration yet.