Decentralized Feature Selection - The Impacts of User-Control over Feature Selection on the Acceptance and Performance of Recommender Systems

How to cite

Zacharias, Jan; Schenk, Alicia von; Klockmann, Victor; Bauer, Kevin. 2022. "Decentralized Feature Selection - The Impacts of User-Control over Feature Selection on the Acceptance and Performance of Recommender Systems." AEA RCT Registry. AEARCTR-0010667. https://doi.org/10.1257/rct.10667-3.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jan Zacharias* , Alicia von Schenk , Victor Klockmann , Kevin Bauer

* Primary investigator

Registry ID
AEARCTR-0010667
DOI
https://doi.org/10.1257/rct.10667-3.0
First registered
Last updated
Status
In development
Trial dates
2023-04-24 – 2023-05-05
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/10667

Abstract

Contemporary artificial intelligence (AI) systems have the potential to benefit both organizations and consumers. A prerequisite for AI systems to materialize their potential is the acceptance and, eventually, usage of these systems by consumers. However, the deployment of AI systems is associated with various challenges, too often leading to resistance and aversion to the algorithms. For instance, consumers oftentimes question the reliability and accuracy of AI systems and, thus, feel misunderstood by them. Literature shows that organizations may take action to reduce algorithm aversion. One possible means presented by prior studies is to allow users to modify AI models and their outcomes. In our work, we investigate how organizations may encounter algorithm aversion by involving users into the prediction-making process. More precisely, we propose an approach where organizations allow users to select which information they pass to the AI system - a process we call decentralized feature selection. In an experimental setting, we investigate the impact of this decentralized feature selection on consumers’ attitudes towards and the actual predictive performance of AI systems.

Intervention

We developed an experimental setting in which participants may
control which variables are submitted to an AI system whose predictions potentially influence the
participants’ payoff. In this experimental setting, participants may choose one binary lottery of a set of 200 binary lotteries which differ in terms of their risk level as well
as objective goodness of the potential payoffs. Participants may leverage a pre-trained AI system,
predicting the participants’ risk attitudes, for preselecting a lottery subset which is aligned with the
individual participants’ risk attitudes. Participants in the treatment group further have the opportunity
to select which attributes the AI system may process; in the following, we refer to this procedure as
decentralized feature selection.

Experimental design

The basic procedure of the experiment is the following: In stage 1, before informing the participants
about subsequent stages, we elicit attributes about participants via a questionnaire. These attributes
serve as input features for the AI-based risk attitude prediction. In stage 2, after the explanation of the
experiment, we present the participants the full list of the 200 possible binary lotteries with censored
payoffs. The lotteries differ in terms of their risk level as well as the goodness of the potential payoffs
(see subsection ‘Details on stage 1’ in the attached file for more information). After the presentation of the full lottery list,
we introduce the AI system to the participants. The AI system is trained to predict participants’ risk
attitudes based on the information provided in the initial questionnaire. Using this prediction, the AI
system recommends the five most suitable lotteries from the full list of 200 lotteries to each individual
participant. In stage 3, we perform the treatment manipulation. In a within-subjects design, we present
participants two different scenarios: One scenario where participants may perform decentralized
feature selection to influence the AI system (treatment condition), and another scenario without
decentralized feature selection, i.e. the AI system uses all participant attributes elicited in the initial
questionnaire (baseline condition). In each scenario, we ask participants to state their willingness to
pay (WTP) for leveraging the respective AI system. In stage 4, participants make their lottery decision.
Participants whose WTP was high enough for receiving the AI support may inspect both the AI-based
lottery recommendations and the full list of 200 lotteries. Participants whose WTP did not reach the
critical BDM-threshold do not receive the AI-based recommendations. In stage 5 we elicit additional
secondary measures as well as participants’ revealed risk attitudes.

Design details

Randomization method
Randomization done automatically by the computer.
Randomization unit
Randomization of treatment and baseline group.
Clusters
We do not plan clustering. Participants stem from Online Platform.
Observations
400
Arms
200 participants per group (treatment vs. baseline).
Primary outcomes
(1) Willingness to pay for usage of the AI system. (2) Performance of the ML model incorporating decentralized feature selection relative to the baseline ML model in terms of established performance metrics (RMSE, Accuracy, ROC AUC, etc.)
Secondary outcomes
(1) Transparency (2) Power (3) Privacy intrusion (4) Curiosity in ML prediction (5) Perceived accuracy of the AI system

Cited by

No citation data available for this registration yet.

Keywords