Reframing Innovation with Generative AI: The Impact of Modified Descriptions on Idea Evaluation
How to cite
Ayoubi, Charles; Just, Julian. 2024. "Reframing Innovation with Generative AI: The Impact of Modified Descriptions on Idea Evaluation." AEA RCT Registry. AEARCTR-0014529. https://doi.org/10.1257/rct.14529-1.1
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This study emphasizes the role of generative AI in modifying the descriptions of innovative solutions and how these modifications influence evaluation outcomes of humans and of AI. Specifically, the research explores how summarization, which condenses solutions into more essential and streamlined versions, and targeted reframing, which changes the writing style and wording for desired purposes, affects human evaluators' perceptions of novelty, feasibility, and other evaluation metrics.
The study's inquiry revolves around AI's ability to reframe ideas by either enhancing or neutralizing the novelty presented in these descriptions. By creating AI-modified summaries, the experiment tests whether these modifications change evaluators' views on the novelty and feasibility of early-stage ideas. The AI-neutralized descriptions allow the team to control for cognitive biases that may arise due to detailed or promotional language. Thus, the study will also investigate differences between human and AI evaluation depending on the summarization and framing.
Key points of consideration include:
Summarization and Cognitive Load: Simplified AI-generated summaries reduce the cognitive burden on evaluators, potentially leading to less biased and more objective assessments.
Neutralization vs. Promotional Framing: The study compares the effects of AI-neutralized descriptions, where promotional language is stripped, versus AI-enhanced versions that promote novelty. It investigates whether these framings impact novelty perception and decision-making differently.
Perceived vs. Objective Novelty: AI-modified descriptions allow a direct comparison between human-perceived novelty, AI-perceived novelty (prompted GPT model), and objective novelty measures (similarity embedding techniques). The research questions whether AI summarization can bridge the gap between these dimensions.
This study thus aims to shed light on how AI-driven text modifications can influence innovation evaluation, providing insights into the role of linguistic framing and summarization in decision-making processes.
Intervention
This study investigates how generative AI-modified descriptions of climate-related solutions impact the evaluation of their novelty, feasibility, and overall potential. The intervention involves creating three versions of each solution: a summary using the same wording and writing style as the original version, an AI-enhanced summary with novelty-promoting language, and an AI-neutralized summary where promotional language is stripped. Master students enrolled in a climate-focused course will evaluate the solutions, randomly receiving solutions from each of the three versions, and full original versions. The aim is to examine how AI-generated summarization and rephrasing affect the evaluators' perceptions and decisions. Each participant will evaluate 15 solutions selected from a pool of 150 versions (50 unique solutions, each in 3 different summary versions) and three original full solutions. Participants will also evaluate non-summarized versions to evaluate the summarization effect. There will be 100 non-summarized solutions (50 original submissions and 50 modified by AI). Each participant will evaluate 4 long solutions (2 originals and 2 modified by AI). For a total of 19 randomized evaluations.
We run a second-round of experimentation in the same empirical context with a simplified version with only two treatments in addition to the control. The treatment consist in two summarized and standardized versions of the control using AI. One long and one short.
Experimental design
This randomized controlled trial (RCT) tests how generative AI-modified descriptions affect the evaluation of climate-related solutions. 100 Master students in two separate institutions will evaluate 19 solutions, randomly selected from 250 versions (50 solutions in 5 variations). Each participant will receive 5 solutions per summarized version type (original summarized, AI-enhanced summarized, AI-neutralized summarized), with the order randomized, and four long versions. The study aims to uncover the effects of AI-modified framing on perceptions of novelty and feasibility.
The second round will consist of 12 solutions to evaluate from 180 versions (60 solutions in 3 conditions).
Design details
Cited by
No citation data available for this registration yet.