Chatbots as Idea Evaluation Sparring Partners

How to cite

Just, Julian. 2025. "Chatbots as Idea Evaluation Sparring Partners." AEA RCT Registry. AEARCTR-0016864. https://doi.org/10.1257/rct.16864-1.3

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Julian Just*

* Primary investigator

Registry ID
AEARCTR-0016864
DOI
https://doi.org/10.1257/rct.16864-1.3
First registered
Last updated
Status
In development
Trial dates
2025-10-03 – 2025-11-17
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/16864

Abstract

Idea evaluation is inherently subjective, often limited by cognitive biases and incomplete information. This study explores how AI chatbots, adopting different conversational stances - such as supportive, critical, balanced perspectives - can broaden evaluators’ judgments. Through controlled interactions, we examine how these AI sparring partners influence assessment confidence, shifts in evaluation, and the overall quality of decisions. By analyzing participants’ personality traits, perceptions of the AI, and their reasoning processes, the research seeks to determine whether AI can effectively expand human perspectives in idea evaluation without distorting judgment. The findings aim to inform the design of human-AI collaboration systems for more informed and consistent decision-making.

Experimental design

We have an experiment with 8 conditions testing how AI sparring partners influence idea evaluation. We may focus
Participants are randomly assigned to one of these setups:

With pre-assessment: Evaluate ideas first, then chat with AI, then re-evaluate. (currently planned as main focus)
Without pre-assessment: Chat with AI first, then evaluate ideas for the first time.

The AI takes one of 4 roles, controlled by a custom GPT system prompt:

Basic (plain/default)
Angel’s Advocate (supportive)
Devil’s Advocate (critical)
Dialectic (balanced debate)

The AI chat is embedded in Qualtrics - participants move seamlessly between survey questions and the GPT conversation.
All conditions measure:
Holistic and detailed idea assessments
Confidence in assessments
Changes in evaluation scores
Alignment with group averages (metadelta)
Convergence of opinions
Personality traits, AI perceptions, and chat transcripts (prompts/rationales).

Design details

Randomization method
Randomization done in office by a computer - Qualtrics
Randomization unit
The unit of randomization is the individual participant.
Clusters
1
Observations
800 evaluations of domain-experts (working in specific field where the to be evalauted idea is from) on Prolific, 150 evalaution of business students - each participant evaluates one idea
Arms
800 prolific experts, 150 students; may be lower if we include some conditions only in a specific sample
Primary outcomes
The key outcome variables in this experiment are idea assessment, assessment confidence, assessment change before and after AI interaction, assessment quality change measured through metadelta, and the convergence of evaluations across participants. Additionally, perceptions of the AI chatbot and unstructured data from prompts and rationales are collected to understand how AI stances influence reasoning and trust.

Cited by

No citation data available for this registration yet.

Keywords