Evaluating Written Arguments in Group Decision-Making: How Payoff Relevance and Decision-Maker Type Shape Argument Characteristics

How to cite

Bechdolf, Mathilde Lea Editha. 2025. "Evaluating Written Arguments in Group Decision-Making: How Payoff Relevance and Decision-Maker Type Shape Argument Characteristics." AEA RCT Registry. AEARCTR-0017247. https://doi.org/10.1257/rct.17247-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Mathilde Lea Editha Bechdolf*

* Primary investigator

Registry ID
AEARCTR-0017247
DOI
https://doi.org/10.1257/rct.17247-1.0
First registered
Last updated
Status
On going
Trial dates
2025-11-14 – 2025-11-28
Countries
Germany (Saxony-Anhalt)
JEL codes
C91, D70, D91, M54
Registry record
http://www.socialscienceregistry.org/trials/17247

Abstract

This study analyzes written arguments that participants submitted in a previous experiment on group decision-making. In the original study, participants were part of groups where a selection mechanism chose one option from several alternatives over 30 rounds. Participants individually ranked three options for each topic according to their preferences (most preferred, second most preferred, and third most preferred) and were invited to individually provide written arguments explaining why their most preferred option should be chosen for the group. Participants' payments depended on whether the chosen option aligned with their preference, with higher payoffs for alignment.
Three different decision-making systems were tested: one where a single person (dictator) made the final decision after reading all arguments and seeing all preferences, one where decisions were made by a voting rule (Borda count) with an observer reading the arguments and seeing preferences, and one where an AI (ChatGPT) made decisions based on arguments and preferences with an observer reading arguments and seeing preferences.
In this follow-up study, we collect independent evaluations of these written arguments. Each argument is rated by 3-4 evaluators who are incentivized for their assessments through additional payment if their rating falls within the median ±1 of all evaluations for that argument on each dimension. Evaluators rate each argument on four characteristics: how emotional it is, how convincing it is, how self-oriented it is (focusing on personal experiences or preferences rather than group or societal ones), and how vivid the language is (using descriptive language and metaphors).
Understanding how people craft arguments under different decision-making systems is important because the perceived relevance of one's voice may influence communication strategies. By analyzing whether arguments differ systematically when they directly influence outcomes versus when they are merely observed, and when the decision-maker is human versus AI, we can gain insights into how institutional design shapes voice expression. These insights will help organizations and policymakers design participatory decision-making processes that effectively incorporate individual voice in group decision contexts.

Intervention

Participants evaluate written arguments from a previous group decision-making experiment. Each argument is independently rated by 3-4 evaluators on four dimensions: emotionality, convincingness, self-orientation, and vividness of language. Evaluators are incentivized through additional payment if their ratings fall within the median ±1 of all ratings for that argument on each dimension.

Experimental design

Independent evaluators rate written arguments from a previous group decision-making experiment. Each argument is rated by 3-4 evaluators on four dimensions using 7-point Likert scales. Evaluators are incentivized for ratings that fall within the median ±1 of all ratings for each argument-dimension combination.

Design details

Randomization method
Individual arguments are the unit of analysis. Each argument is randomly assigned to 3-4 evaluators. The order in which arguments are presented to evaluators is also randomized at the individual evaluator level.
Randomization unit
Individual arguments are the unit of analysis. Each argument is randomly assigned to 3-4 evaluators. The order in which arguments are presented to evaluators is also randomized at the individual evaluator level.
Clusters
Not applicable - no clustering design
Observations
5,627 arguments evaluated by approximately 212 evaluators, resulting in 16,960 total evaluations (each argument rated 3-4 times, each evaluator rates 80 arguments).
Arms
Dictator: 1934 arguments Borda: 1929 arguments AI: 1764 arguments Each argument is evaluated 3-4 times by independent raters, with each rater evaluating 80 arguments. Total of approximately 212 evaluators providing 16,960 evaluations.
Primary outcomes
1. Emotionality ratings of arguments (7-point Likert scale) 2. Convincingness ratings of arguments (7-point Likert scale) 3. Self-orientation ratings of arguments (7-point Likert scale) 4. Vividness ratings of arguments (7-point Likert scale)
Secondary outcomes
1. Inter-rater agreement/reliability measures (ICC) 2. Evaluator characteristics (demographics) 3. Relationship between argument length and ratings

Cited by

No citation data available for this registration yet.

Keywords