AI Advice Depth and Breadth in Technical Customer Support
How to cite
Just, Julian. 2025. "AI Advice Depth and Breadth in Technical Customer Support." AEA RCT Registry. AEARCTR-0015965. https://doi.org/10.1257/rct.15965-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This study investigates the impact of integrating AI advice on technical customer support, a critical and resource-intensive function that minimizes delays and costs. The field experiment will be conducted with 54 technicians from a technical customer support team solving approximately 180 problems weekly, focusing on their problem-solving efficiency and effectiveness. Technicians currently face limitations in accessing stored knowledge and prefer varied advice styles, leading to unclear impacts on their problem-solving performance.
The research aims to demonstrate how AI advice influences problem-solving compared to existing practices. It involves two subexperiments in the field. In the first one, it examines different two AI chatbot response styles - concise and detailed - in a general chat not directly related to a specific problem and their effects on technician satisfaction, working modes and learning. In a second one, it examines four different chatbot settings - one specific or three suggestions based on two different information retrieval options - and their effects on resolution time, technician satisfaction, resolution quality, working modes and learning. The study further considers different problem types and technicians' backgrounds.
The interventions build on an AI system that integrates knowledge from internal databases, e.g., previously resolved incident, manuals, wikis, using retrieval-augmented generation. In the first experiment, via the system prompt two different AI chatbots that vary in their information depth - concise and detailed - will be randomly be available for each problem-solving task. The second experiment, directly connects problems to the internal databases and initiates the chat with suggestions of potential solutions randomly generated.
Previous studies in the same context and feedback from technicians brought up unclear implications on which AI advice style will be more helpful and effective. We expect different impacts on resolution time, chatbot interaction patterns and working modes depending on the collaboration with the different chatbot versions. We also expect different impact depending on the problem type and technician's experience.
Intervention
The intervention builds on an AI system that integrates knowledge from internal databases, e.g., previously resolved incident, manuals, wikis, using retrieval-augmented generation. In the first field experiment, via the system prompt two different AI chatbots - concise or verbose - will be randomly be available for each problem-solving task. The second experiment, inserted problems are directly connected to the internal databases and through four different inititating prompts different suggestion types for potential solutions are randomly generated.
Experimental design
The experiment introduces an AI chatbot that provides real-time advice while technicians resolve incoming problems. The chatbot offers two different ways to obtain problem-solving support:
1. General knowledge chat unrelated to a specific problem: Advice styles - concise or verbose (depth) - and integrates knowledge from internal databases. Technicians will be asked to evaluate the quality of the advice, as well as share information on the problem-solving processess ex-post. In each chat they will be randomly assigned to one AI chatbot condition.
2. Problem-specific chat linked to a specific problem: Initial suggestions with for different styles (2x2) - either one suggestions or three (breadth of alternatives), either based on information just from the system where the error occured or from all related systems (breadth of information retrieved). Technicians will be asked to evaluate the quality of the advice, as well as share information on the problem-solving processess ex-post. Depending on the problem-id they will be randomly assigned to one initial AI chatbot suggestion condition.
Design details
Cited by
No citation data available for this registration yet.