The Impact of Explanatory Strategies on Employee Trust in Algorithmic Forecasts: An Experimental Investigation

How to cite

Röder, Andreas; Seiter, Mischa. 2024. "The Impact of Explanatory Strategies on Employee Trust in Algorithmic Forecasts: An Experimental Investigation." AEA RCT Registry. AEARCTR-0014040. https://doi.org/10.1257/rct.14040-3.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Andreas Röder* , Mischa Seiter

* Primary investigator

Registry ID
AEARCTR-0014040
DOI
https://doi.org/10.1257/rct.14040-3.0
First registered
Last updated
Status
In development
Trial dates
2025-03-11 – 2025-03-22
Countries
Private
JEL codes
C91, D91, M41
Registry record
http://www.socialscienceregistry.org/trials/14040

Abstract

Algorithmically generated information is playing an increasingly central role in managers' decision-making processes. However, the value of these insights is only fully realized when they are effectively utilized. Consequently, the literature on algorithm aversion has explored the factors that influence their adoption. One underexplored phenomenon is managers' tendency to seek reassurance from their superiors. In our experiment, we investigate whether such advice-seeking behavior occurs and whether it can be influenced by providing explanations through explainable AI (XAI). We employ an online experiment involving crowd workers to isolate the effects of different XAI methods (Feature Importance Explanations vs. Counterfactual Explanations) on advice-seeking behavior.

Intervention

Participants assume the role of a store manager in a convenience store chain. They are tasked with forecasting sales of the upcoming month using a sales development dashboard, which is similar to contemporary business intelligence tools. The assumed store chain uses AI to assist the store manager in the forecasting task. We implement two different explanatory strategies i) Feature Importance Explanations and ii) Counterfactual Explanations along the AI forecasts for users. During the experiment, participants can interact with the sales development dashboard to learn about the store using the information provided by the system, the AI prediction and the explanation associated with their experimental condition.

Experimental design

Participants will be setting sales targets for a fictious convenience store. They will be randomly divided into three experimental conditions, each subject to different conditions formulated by our independent variable:

• Algorithmic Advice (AI vs. AI & Feature Importance Explanation vs. AI & Counterfactual Explanation)

On the welcome page of the experiment, we will inform participants about the procedure and potential exclusions through attention checks. They are instructed to immerse themselves into the situation of setting targets for a convenience store. To further incentivize participants, they receive a bonus for setting accurate sales targets.

The experiment can be sectioned into the following steps:
1. Participants receive general information of their store and the stores past sales performance.
2. Participants indicate their sales expectations for the upcoming month.
3. Participants are provided an AI forecast (and a corresponding explanation depending on the experimental condition) of expected sales for the upcoming month.
4. Based on the information provided beforehand, participants must set a sales target for their store.
5. The participants can choose to ask their supervisor for advice. However, advice is costly and reduces their potential bonus payment
6. Participants can readjust their previously selected sales target (based on the information provided by their supervisor).

After completing the target setting process, participants will be asked to answer a post experimental questionnaire, to query their experience and general preferences. Finally, participants are asked to provide sociodemographic information.

Design details

Randomization method
Participants will be randomly assigned to the experimental conditions by a designated function of the website.
Randomization unit
Randomization will be done at the participant level
Clusters
-
Observations
Our sample size will be calculated based on the results of a pre-test using Stata. Expected sample size: 540 participants.
Arms
-
Primary outcomes
Advice seeking from supervisors
Secondary outcomes
Weight of AI advice

Cited by

No citation data available for this registration yet.

Keywords