Reframing Innovation with Generative AI: The Impact of Modified Descriptions on Idea Evaluation

How to cite

Ayoubi, Charles; Just, Julian. 2024. "Reframing Innovation with Generative AI: The Impact of Modified Descriptions on Idea Evaluation." AEA RCT Registry. AEARCTR-0014529. https://doi.org/10.1257/rct.14529-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Charles Ayoubi* , Julian Just

* Primary investigator

Registry ID
AEARCTR-0014529
DOI
https://doi.org/10.1257/rct.14529-1.1
First registered
Last updated
Status
In development
Trial dates
2024-11-19 – 2025-11-30
Countries
Private
JEL codes
O31, O32, C93, D8
Registry record
http://www.socialscienceregistry.org/trials/14529

Abstract

This study emphasizes the role of generative AI in modifying the descriptions of innovative solutions and how these modifications influence evaluation outcomes of humans and of AI. Specifically, the research explores how summarization, which condenses solutions into more essential and streamlined versions, and targeted reframing, which changes the writing style and wording for desired purposes, affects human evaluators' perceptions of novelty, feasibility, and other evaluation metrics.
The study's inquiry revolves around AI's ability to reframe ideas by either enhancing or neutralizing the novelty presented in these descriptions. By creating AI-modified summaries, the experiment tests whether these modifications change evaluators' views on the novelty and feasibility of early-stage ideas. The AI-neutralized descriptions allow the team to control for cognitive biases that may arise due to detailed or promotional language. Thus, the study will also investigate differences between human and AI evaluation depending on the summarization and framing.

Key points of consideration include:
Summarization and Cognitive Load: Simplified AI-generated summaries reduce the cognitive burden on evaluators, potentially leading to less biased and more objective assessments.
Neutralization vs. Promotional Framing: The study compares the effects of AI-neutralized descriptions, where promotional language is stripped, versus AI-enhanced versions that promote novelty. It investigates whether these framings impact novelty perception and decision-making differently.
Perceived vs. Objective Novelty: AI-modified descriptions allow a direct comparison between human-perceived novelty, AI-perceived novelty (prompted GPT model), and objective novelty measures (similarity embedding techniques). The research questions whether AI summarization can bridge the gap between these dimensions.

This study thus aims to shed light on how AI-driven text modifications can influence innovation evaluation, providing insights into the role of linguistic framing and summarization in decision-making processes.

Intervention

This study investigates how generative AI-modified descriptions of climate-related solutions impact the evaluation of their novelty, feasibility, and overall potential. The intervention involves creating three versions of each solution: a summary using the same wording and writing style as the original version, an AI-enhanced summary with novelty-promoting language, and an AI-neutralized summary where promotional language is stripped. Master students enrolled in a climate-focused course will evaluate the solutions, randomly receiving solutions from each of the three versions, and full original versions. The aim is to examine how AI-generated summarization and rephrasing affect the evaluators' perceptions and decisions. Each participant will evaluate 15 solutions selected from a pool of 150 versions (50 unique solutions, each in 3 different summary versions) and three original full solutions. Participants will also evaluate non-summarized versions to evaluate the summarization effect. There will be 100 non-summarized solutions (50 original submissions and 50 modified by AI). Each participant will evaluate 4 long solutions (2 originals and 2 modified by AI). For a total of 19 randomized evaluations.

We run a second-round of experimentation in the same empirical context with a simplified version with only two treatments in addition to the control. The treatment consist in two summarized and standardized versions of the control using AI. One long and one short.

Experimental design

This randomized controlled trial (RCT) tests how generative AI-modified descriptions affect the evaluation of climate-related solutions. 100 Master students in two separate institutions will evaluate 19 solutions, randomly selected from 250 versions (50 solutions in 5 variations). Each participant will receive 5 solutions per summarized version type (original summarized, AI-enhanced summarized, AI-neutralized summarized), with the order randomized, and four long versions. The study aims to uncover the effects of AI-modified framing on perceptions of novelty and feasibility.

The second round will consist of 12 solutions to evaluate from 180 versions (60 solutions in 3 conditions).

Design details

Randomization method
Randomization will be done by a computer algorithm from Qualtrics that assigns each participant to a random selection of solutions and randomizes the order in which the solutions are presented.
Randomization unit
The unit of randomization is at the individual level, where each participant receives 19 randomly selected solutions in randomized order across the three treatment types (original summary, AI-enhanced summary, and AI-neutralized summary) and the four long full versions. There is no clustering.
Clusters
2 For the second round. Two schools as well as a group of experts in sustainability.
Observations
The planned number of observations is 1900 evaluations (100 students x 19 evaluations per student).
Arms
Each participant will evaluate 19 solutions, distributed across the three treatment arms and the original full version: Original summary version: ~500 evaluations AI-enhanced summary version: ~500 evaluations AI-neutralized summary version: ~500 evaluations Original full version: ~200 evaluations Modified full version: ~200 evaluations Second round: Original version: ~240 observations (4 solutions *60 participants) AI-summarized-long: ~240 observations (4 solutions *60 participants) AI-summarized-short: ~240 observations (4 solutions *60 participants)
Primary outcomes
Potential for Impact: The planned solution implementation has the potential to impact the intended population. Feasibility: The solution suggests a realistic, practical plan for being implemented, and it is feasible in the given context. Innovative Approach: The solution includes a new technology, a new application of technology, a new business model, or a new process for solving the Challenge.
Secondary outcomes
Bias Detection: Whether summarization and/or AI-modified descriptions (enhanced or neutralized) increase or decrease evaluators’ biases toward novelty or feasibility. Participants will also be asked to judge the: Clarity about solution: After reading the description of the solution above, you have a clear idea of what the solution is about. Ease of readability: How easy was the process of reading the solution description.  Consistency in Evaluation: The variance in evaluators' responses when exposed to AI-modified vs. original descriptions. Comparison with AI Evaluation: How human evaluation of novelty compares to the novelty scores generated by the AI system.

Cited by

No citation data available for this registration yet.

Keywords