Chatbots as Idea Evaluation Sparring Partners
How to cite
Just, Julian. 2025. "Chatbots as Idea Evaluation Sparring Partners." AEA RCT Registry. AEARCTR-0016864. https://doi.org/10.1257/rct.16864-1.3
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Idea evaluation is inherently subjective, often limited by cognitive biases and incomplete information. This study explores how AI chatbots, adopting different conversational stances - such as supportive, critical, balanced perspectives - can broaden evaluators’ judgments. Through controlled interactions, we examine how these AI sparring partners influence assessment confidence, shifts in evaluation, and the overall quality of decisions. By analyzing participants’ personality traits, perceptions of the AI, and their reasoning processes, the research seeks to determine whether AI can effectively expand human perspectives in idea evaluation without distorting judgment. The findings aim to inform the design of human-AI collaboration systems for more informed and consistent decision-making.
Experimental design
We have an experiment with 8 conditions testing how AI sparring partners influence idea evaluation. We may focus
Participants are randomly assigned to one of these setups:
With pre-assessment: Evaluate ideas first, then chat with AI, then re-evaluate. (currently planned as main focus)
Without pre-assessment: Chat with AI first, then evaluate ideas for the first time.
The AI takes one of 4 roles, controlled by a custom GPT system prompt:
Basic (plain/default)
Angel’s Advocate (supportive)
Devil’s Advocate (critical)
Dialectic (balanced debate)
The AI chat is embedded in Qualtrics - participants move seamlessly between survey questions and the GPT conversation.
All conditions measure:
Holistic and detailed idea assessments
Confidence in assessments
Changes in evaluation scores
Alignment with group averages (metadelta)
Convergence of opinions
Personality traits, AI perceptions, and chat transcripts (prompts/rationales).
Design details
Cited by
No citation data available for this registration yet.