Welfare Gains of Online Rating Systems
How to cite
Bohren, Noah; Hakimov, Rustamdjan; Pinto, Luis Santos. 2023. "Welfare Gains of Online Rating Systems." AEA RCT Registry. AEARCTR-0011386. https://doi.org/10.1257/rct.11386-2.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We study the efficacy of rating systems, similar to the one used in online marketplaces. In this study, we investigate the effectiveness of rating systems in assisting consumers with product selection, focusing on both vertical and horizontal markets. Additionally, we explore the impact of grouping ratings and implementing freezing periods on welfare in horizontally differentiated markets. Our research aims to answer the following questions:
1. Does the introduction of a rating system increase welfare in vertically differentiated markets?
2. Does the introduction of a rating system increase welfare in horizontally differentiated markets?
3. Does providing average reviews broken down by relevant individual characteristics (filtering) increase welfare in horizontally differentiated markets?
4. Does the introduction of a “freezing period” (the average rating is disclosed only when a minimum amount of ratings are available) improve welfare?
***************
New treatments for November 2024:
AT1: Does the introduction of an algorithm based recommender system increase welfare in horizontally differentiated markets? How does it compare to filtering?
AT2: Does the combination of a "classical" rating system (as implemented in the earlier treatments) and the algorithm based recommender system increase welfare in the horizontal market and does the effect decrease over time?
Intervention
## Sketch of Design:
We conduct an online experiment on the Prolific platform to study the efficacy of various online rating systems in vertically and horizontally differentiated markets.
We create artificial marketplaces where the goods participants can buy are tasks that they can perform. The tasks vary and determine whether a market is vertically or horizontally differentiated. Each participant is endowed with £2.5 and decides whether to buy one of two tasks or not. The two tasks are quizzes about celebrities that consist of 10 questions with 6 possible answers. Each task lasts 110 seconds. Participation in Task 1 and 2 costs £1.7 but rewards £0.45 per correct answer. If they choose not to buy a task, participants spend 110 seconds counting zeros in the matrices task, with no incentives.
We measure welfare as the average monetary outcome of participants.
Vertically Differentiated Markets:
In vertically differentiated markets, Task 1 is set such that it is easier than Task 2 for most participants. We call Task 1 “easy” and Task 2 “hard”.
e.g. *“Among all elected US American presidents, what was the name of the first who had African origin?”* VS *“Which of those actors won the most Oscars for acting?”*
Horizontally Differentiated Markets:
In horizontally differentiated markets, Task 1 is set such that it is easier than Task 2 for only half of the participants. To generate such variation, we recruit two types of participants: people between 18-30 years old and 50+ years old. We call Task 1 “young” and Task 2 “old”.
e.g. *"Who is the celebrity with the most subscribers on YouTube?”* VS *"In which TV- show did the character of JR Ewing appear?”*
If participants buy a task, once they complete it, participants have the possibility (but not the obligation) to give a rating (1 to 5 stars) by answering the following question: *"Please give us your opinion on the task you just participated in. This information can be helpful for future participants”*
We create 6 treatments that vary the type of market (vertical or horizontal) and which rating system is used (none, standard, filtered, frozen)
### Key stages
We define two key stages of the experiment.
1. At the **buying stage,** participants are informed of:
(a) The prices of each task as well as the payment scheme.
(b) A description of each task. Tasks 1 and 2 are both described as a quiz on celebrities. The outside option of not buying is described as having to count zeros in tables.
(c) Average (and number) of ratings for Task 1 and 2. (Whether this information is available or not and how it is disclosed depends on treatments)
2. At the **rating stage,** participants of Task 1 and 2 receive their scores and have an opportunity to rate the tasks.
An important consideration in our design is the evolution of ratings. When ratings are available during the buying stage (all but the baseline treatments), the observations become interdependent. This means that early ratings might influence the entire path of rating developments. To ensure several independent histories of ratings, we create 14 groups per treatment. In each group, 30 participants arrive sequentially and observe the ratings of the previous participants. Having independent groups allows us to observe potential different paths of rating evolution, which, in turn, allows us to measure whether there is a large variance in the welfare gains in the markets.
****************
Additional treatments: November 2024
To further explore the effectiveness of different systems in horizontally differentiated markets, we introduce two additional treatments: an algorithm-based recommendation and a combined rating and algorithm system. These treatments build on the existing experimental framework to examine how personalized information and rating systems interact in shaping welfare outcomes.
Experimental design
### Treatments
We create two **baseline** treatments. One vertical (Task 1 is easy and Task 2 is hard) and one horizontal (Task 1 is young and Task 2 is old). For each, we recruit 200 participants who select a task and give a rating. The average rating from previous participants is never disclosed. This allows us to create a baseline and see how much welfare participants are able to generate **without** a rating system. (Note, we do not need participants to arrive sequentially since ratings are never displayed)
We then create two **rating** treatments where the average rating from previous participants is available at the buying stage. For the reasons described above, for each treatment, we generate 14 groups of 30 participants who enter the marketplace sequentially and where the average ratings are updated at every passing. This allows us to see how much additional welfare is generated by the introduction of a rating system.
We believe that rating systems are less effective when goods are horizontally differentiated. For this reason, we study two variations of rating systems that are used by some online platforms and could help improve welfare for the horizontal markets.
1. Filtering: This treatment is similar to “Rating Horizontal” but rather than displaying the average (and number) of ratings from previous participants, it is the average (and number) of ratings for each type (18-30 and 50+) that is displayed.
2. Freezing: This treatment is similar to “Rating Horizontal” but each task needs at least 5 ratings for the average (and number) or ratings to be disclosed.
### Additional exercises
In addition, for all participants and before the buying stage, we measure self-confidence in performance in celebrity quizzes. Participants have to answer the following question: *Imagine taking a quiz about celebrities. Out of 100 randomly selected people who also took the same quiz, how many do you think would perform worse than you?* This allows us to explore the effect of initial expectations on buying decisions and ratings.
Finally, we also measure risk aversion by asking the following “Are you generally a person who is fully prepared to take risks or do you try to avoid taking risks? [Scale from 0 to 10]]”.
**********************
## Additional treatments: November 2024
#Treatment 1: Algorithm-Based Recommendation
In this treatment, we introduce an ‘algorithm’ designed to recommend the most profitable option based on participants’ individual characteristics. The algorithm is trained on data from previous participants in this experiment. The algorithm is very simple as it will suggest participants to buy the age matching task (i.e. task old for old participants and task young for young participants). The algorithm’s suggestion will be presented as: ”Based on previous performance of participants similar to you, we suggest you buy task [age group].”
#Procedure:
Participants first complete the same introductory questionnaire as in previous treatments (including demographic information and confidence). During the buying stage, they are presented with the algorithm’s recommendation, which is tailored to their specific characteristics. Participants then choose a task or opt to count zeros, complete the chosen task, rate it, and exit the experiment.
#Sample:
We recruit 200 participants for this treatment (100 old and 100 young). Each
participant makes a decision independently (without sequential interdependence of
ratings).
#Treatment 2: Combined Rating and Algorithm-
This treatment combines the existing ”Rating Horizontal” system with the algorithm-based recommendation. Participants are provided with both the average ratings of previous participants (without filtering) and the algorithm’s recommendation during the buying stage.
#Procedure:
As in previous treatments, participants answer the introductory questionnaire and are exposed to the buying stage, where they see both the algorithm’s recommendation and the average rating (along with the number of ratings) from previous participants. After making their choice, they complete the task, rate it, and exit the experiment.
This treatment follows the same 14x30 sequential structure as in the ”Rating Horizontal” treatment, with ratings evolving across groups of participants. In each sequence, 15 old and 15 young participants will be randomly assigned a position in the sequence.
#Sample:
We recruit 14 groups of 30 participants (total 420 participants) who enter the
marketplace sequentially and observe the ratings of previous participants.
Design details
Cited by
No citation data available for this registration yet.