Evaluating the impact of AI-powered reasoning in deliberative processes

How to cite

Enríquez, José Ramón; Braley, Alia; Chen, Nuole; Pei, Jiaxin; Pentland, Alex; Stockinger, Elisabeth; Tsai, Lily. 2025. "Evaluating the impact of AI-powered reasoning in deliberative processes." AEA RCT Registry. AEARCTR-0015461. https://doi.org/10.1257/rct.15461-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

José Ramón Enríquez* , Alia Braley , Nuole Chen , Jiaxin Pei , Alex Pentland , Elisabeth Stockinger , Lily Tsai

* Primary investigator

Registry ID
AEARCTR-0015461
DOI
https://doi.org/10.1257/rct.15461-2.0
First registered
Last updated
Status
In development
Trial dates
2025-02-01 – 2025-12-31
Countries
Private
JEL codes
C91, D78
Registry record
http://www.socialscienceregistry.org/trials/15461

Abstract

This pre-analysis plan describes an experimental study evaluating whether AI-powered reasoning interventions and monetary incentives can enhance argument quality in an online policy debate on mental illness and U.S. gun regulation. Participants will be randomly assigned to either a reasoning (treatment) condition or a control condition, with further cross-randomization of direct or lottery-based financial rewards for thoughtful contributions. The primary outcomes include measures of argument thoughtfulness and compromise. In addition, secondary analyses will examine potential mechanisms, such as cognitive reflection, emotional state, diversity of perspectives, and engagement. This study seeks to identify practical, scalable methods for improving deliberative discourse in digital contexts.

Intervention

We test two main interventions for improving argument quality in an online policy debate on mental illness and firearm regulation in the United States:

1) AI-powered Reasoning: Participants receive prompts from a large language model (LLM) designed to foster more reflective, logically coherent arguments (i.e., Socratic dialogue), solo guided reflection (i.e., Reflective paragraph); LLM-powered emotional regulation (i.e., Emotional-regulation LLM); grammar correction (i.e., Grammar-correction LLM), or waiting control.

2) Monetary Incentives: Participants receive the option to receive direct monetary bonus (for top quartile arguments), a lottery-based bonus (for top quartile arguments), or no incentive.

Experimental design

We conduct an online experiment with participants recruited to debate whether people with mental illnesses should be allowed to purchase firearms. We randomize at the individual level along two dimensions:
(i) reasoning interventions (Socratic dialogue LLM vs. Reflective paragraph vs. Emotional-regulation LLM vs. Grammar-correction LLM vs. waiting control) and
(ii) monetary incentives (direct bonus vs. lottery vs. none).

The primary objective is to determine which intervention(s) most effectively increase argument thoughtfulness and willingness to compromise. We measure outcomes post-treatment via rubric-based coding of each participant’s main argument, changes in stance, and secondary indicators like voting and donation behavior.

Final analyses will compare each treatment arm with pooled or disaggregated controls, depending on tests of equivalence among control conditions.

Details in Pre-Analysis Plan attached.

Design details

Randomization method
Pseudo-random generator within the survey/intervention platform (deliberation.io). Each participant has equal probability of assignment to each cell in the 5×3 factorial design.
Randomization unit
Individual.
Clusters
N/A.
Observations
Initial intervention: 1,500 when analyzing individual-level outcomes. Up to 18,000 (1500 x 12) when analyzing individual-vote level outcomes. Medium-term effects and cross-topic intervention: 4,000 when individual-level outcomes. Up to 36,000 (4,000 x 9) when analyzing individual-vote level outcomes.
Arms
1500: 300 for each reasoning condition; 500 for each monetary incentive condition. 4000: 2000 for each reasoning condition.
Primary outcomes
1. Argument thoughtfulness score: A composite measure (0–6 scale) reflecting logical structure, clarity, and depth of each participant’s main comment. 2. Position shifting or compromise: The extent to which participants move from an extreme stance toward a middle-ground position on the policy.
Secondary outcomes
1. Cognitive Reflection: Changes in CRT scores (pre-post). 2. Emotional State: Pre-post changes on a 2-dimension affect scale. 3. Perspective Diversity: Type-token ratio, semantic similarity variance, voting weighted average. 4. Engagement Indices: Number of comments edited or length of comment, voting behavior, willingness to donate, time spent.

Cited by

No citation data available for this registration yet.

Keywords