Evaluating Written Arguments in Group Decision-Making: How Payoff Relevance and Decision-Maker Type Shape Argument Characteristics
How to cite
Bechdolf, Mathilde Lea Editha. 2025. "Evaluating Written Arguments in Group Decision-Making: How Payoff Relevance and Decision-Maker Type Shape Argument Characteristics." AEA RCT Registry. AEARCTR-0017247. https://doi.org/10.1257/rct.17247-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This study analyzes written arguments that participants submitted in a previous experiment on group decision-making. In the original study, participants were part of groups where a selection mechanism chose one option from several alternatives over 30 rounds. Participants individually ranked three options for each topic according to their preferences (most preferred, second most preferred, and third most preferred) and were invited to individually provide written arguments explaining why their most preferred option should be chosen for the group. Participants' payments depended on whether the chosen option aligned with their preference, with higher payoffs for alignment.
Three different decision-making systems were tested: one where a single person (dictator) made the final decision after reading all arguments and seeing all preferences, one where decisions were made by a voting rule (Borda count) with an observer reading the arguments and seeing preferences, and one where an AI (ChatGPT) made decisions based on arguments and preferences with an observer reading arguments and seeing preferences.
In this follow-up study, we collect independent evaluations of these written arguments. Each argument is rated by 3-4 evaluators who are incentivized for their assessments through additional payment if their rating falls within the median ±1 of all evaluations for that argument on each dimension. Evaluators rate each argument on four characteristics: how emotional it is, how convincing it is, how self-oriented it is (focusing on personal experiences or preferences rather than group or societal ones), and how vivid the language is (using descriptive language and metaphors).
Understanding how people craft arguments under different decision-making systems is important because the perceived relevance of one's voice may influence communication strategies. By analyzing whether arguments differ systematically when they directly influence outcomes versus when they are merely observed, and when the decision-maker is human versus AI, we can gain insights into how institutional design shapes voice expression. These insights will help organizations and policymakers design participatory decision-making processes that effectively incorporate individual voice in group decision contexts.
Intervention
Participants evaluate written arguments from a previous group decision-making experiment. Each argument is independently rated by 3-4 evaluators on four dimensions: emotionality, convincingness, self-orientation, and vividness of language. Evaluators are incentivized through additional payment if their ratings fall within the median ±1 of all ratings for that argument on each dimension.
Experimental design
Independent evaluators rate written arguments from a previous group decision-making experiment. Each argument is rated by 3-4 evaluators on four dimensions using 7-point Likert scales. Evaluators are incentivized for ratings that fall within the median ±1 of all ratings for each argument-dimension combination.
Design details
Cited by
No citation data available for this registration yet.