Evaluating the impact of AI-powered reasoning in deliberative processes
How to cite
Enríquez, José Ramón; Braley, Alia; Chen, Nuole; Pei, Jiaxin; Pentland, Alex; Stockinger, Elisabeth; Tsai, Lily. 2025. "Evaluating the impact of AI-powered reasoning in deliberative processes." AEA RCT Registry. AEARCTR-0015461. https://doi.org/10.1257/rct.15461-2.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This pre-analysis plan describes an experimental study evaluating whether AI-powered reasoning interventions and monetary incentives can enhance argument quality in an online policy debate on mental illness and U.S. gun regulation. Participants will be randomly assigned to either a reasoning (treatment) condition or a control condition, with further cross-randomization of direct or lottery-based financial rewards for thoughtful contributions. The primary outcomes include measures of argument thoughtfulness and compromise. In addition, secondary analyses will examine potential mechanisms, such as cognitive reflection, emotional state, diversity of perspectives, and engagement. This study seeks to identify practical, scalable methods for improving deliberative discourse in digital contexts.
Intervention
We test two main interventions for improving argument quality in an online policy debate on mental illness and firearm regulation in the United States:
1) AI-powered Reasoning: Participants receive prompts from a large language model (LLM) designed to foster more reflective, logically coherent arguments (i.e., Socratic dialogue), solo guided reflection (i.e., Reflective paragraph); LLM-powered emotional regulation (i.e., Emotional-regulation LLM); grammar correction (i.e., Grammar-correction LLM), or waiting control.
2) Monetary Incentives: Participants receive the option to receive direct monetary bonus (for top quartile arguments), a lottery-based bonus (for top quartile arguments), or no incentive.
Experimental design
We conduct an online experiment with participants recruited to debate whether people with mental illnesses should be allowed to purchase firearms. We randomize at the individual level along two dimensions:
(i) reasoning interventions (Socratic dialogue LLM vs. Reflective paragraph vs. Emotional-regulation LLM vs. Grammar-correction LLM vs. waiting control) and
(ii) monetary incentives (direct bonus vs. lottery vs. none).
The primary objective is to determine which intervention(s) most effectively increase argument thoughtfulness and willingness to compromise. We measure outcomes post-treatment via rubric-based coding of each participant’s main argument, changes in stance, and secondary indicators like voting and donation behavior.
Final analyses will compare each treatment arm with pooled or disaggregated controls, depending on tests of equivalence among control conditions.
Details in Pre-Analysis Plan attached.
Design details
Cited by
No citation data available for this registration yet.