Accounting for Heuristics in Reputation Systems: An Interdisciplinary Approach on Aggregation Processes

How to cite

Straaten, Dirk van; Djawadi, Behnud. 2018. "Accounting for Heuristics in Reputation Systems: An Interdisciplinary Approach on Aggregation Processes." AEA RCT Registry. AEARCTR-0003656. https://doi.org/10.1257/rct.3656-2.5

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Dirk van Straaten* , Behnud Djawadi

* Primary investigator

Registry ID
AEARCTR-0003656
DOI
https://doi.org/10.1257/rct.3656-2.5
First registered
Last updated
Status
On going
Trial dates
2016-12-19 – 2023-12-31
Countries
Private
JEL codes
D12 D81 C91
Registry record
http://www.socialscienceregistry.org/trials/3656

Abstract

Aggregation metrics in reputation systems are important for overcoming information overload. When using
these metrics, technical aggregation functions such as the arithmetic mean are implemented to measure the
valence of product ratings. However, it is unclear whether the implemented aggregation functions match the
inherent aggregation patterns of customers. In our experiment, we elicit customers’ aggregation heuristics
and contrast these with reference functions. Our findings indicate that, overall, the arithmetic mean performs
best in comparison with other aggregation functions. However, our analysis on an individual level reveals
heterogeneous aggregation patterns. Major clusters exhibit a binary bias (i.e., an over-weighting of moderate
ratings and under-weighting of extreme ratings) in combination with the arithmetic mean. Minor clusters
focus on 1-star ratings or negative (i.e., 1-star and 2-star) ratings. Thereby, inherent aggregation patterns
are neither affected by variation of provided information nor by individual characteristics such as experience,
risk attitudes, or demographics.

Experimental design

We conduct a randomized controlled trial with a control group (baseline). In both groups, we run twelve periods containing three customer rating distributions each. For the purpose of incentivization we use real customer rating distributions in three periods whereby participants have the chance to win the associated products. These customer rating distributions have the unintended characteristic that at least one stochastically dominates at least one distribution out of this decision set. In the other periods subjects rank artificial customer rating distributions that have the feature of no (first-order) stochastic dominance.
Since it is crucial for our machine learning approach and its underlying assumptions,we plan to exclude decisions that contain stochastically dominated customer rating distributions for this part of analysis.

Design details

Randomization method
Randomization of the decision sets is done by a computer. E.g., decision set A is ranked by participant x in period k and by participant y in period f. In addition, the order the distributions are displayed (i.e., left, middle, right) is randomized by the computer.
Randomization unit
individual
Clusters
2 treatments: baseline (only customer rating distributions displayed) and information treatment (additionally, relative frequencies and the arithmetic mean displayed)
Observations
112 participants
Arms
56 participants per treatment
Primary outcomes
Ranking of customer rating distributions

Cited by

No citation data available for this registration yet.

Keywords