Accepted with Revisions: Evaluating AI-Assisted Scientific Writing

How to cite

Hazra, Sanchaita; Lee, Doeun; Majumder, Bodhisattwa Prasad; Kumar, Sachin. 2025. "Accepted with Revisions: Evaluating AI-Assisted Scientific Writing." AEA RCT Registry. AEARCTR-0016740. https://doi.org/10.1257/rct.16740-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Sanchaita Hazra* , Doeun Lee , Bodhisattwa Prasad Majumder , Sachin Kumar

* Primary investigator

Registry ID
AEARCTR-0016740
DOI
https://doi.org/10.1257/rct.16740-1.0
First registered
Last updated
Status
On going
Trial dates
2025-01-01 – 2026-06-30
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/16740

Abstract

Large Language Models (LLMs) have seen expanding application across domains, yet their effectiveness as assistive tools for scientific writing—an endeavor requiring precision, multimodal synthesis, and domain expertise—remains insufficiently evaluated. We examine the potential of LLMs to support domain experts in scientific writing, with a focus on abstract composition. We design an incentivized randomised trial with a hypothetical conference set up where participants with relevant expertise are segregated into authors and reviewers. Inspired by methods in behavioral science, our novel incentive structure encourages participants to produce high-quality outputs. Authors edit original (control) or their AI-generated (treatment) abstracts of published research from top-tier conferences. Reviewers evaluate if the edited abstract provides adequate justice to the research presented in the original abstract.

Intervention

We simulate a realistic writing environment where authors (domain experts), the intended users of an AI assistant, use and edit scientific abstracts to create scientific prose to get it accepted by atleast two reviewers. We have a 2x2 design for the Author study. The first dimension includes the source of the abstract: original or AI-generated. The second dimension includes information disclosure or not about the source of the abstracts. No treatments for Reviewers who review the edited abstract in reference to the original abstract and answer if the edited abstract provides adequate justice to the research idea presented in the original abstract.

Experimental design

We simulate a realistic writing environment where authors (domain experts), the intended users of an AI assistant, use and edit scientific abstracts to create scientific prose to get it accepted by atleast two reviewers. We have a 2x2 design for the Author study. The first dimension includes the source of the abstract: original or AI-generated. The second dimension includes information disclosure or not about the source of the abstracts. No treatments for Reviewers who review the edited abstract in reference to the original abstract and answer if the edited abstract provides adequate justice to the research idea presented in the original abstract. We collect secondary beliefs like confidence and readability of abstracts.

Design details

Randomization method
Authors: One of the three edited abstracts will be randomly selected for bonus payments. Reviewers: One of the twenty comparison pairs will be randomly selected for bonus payments. All randomizations by a random number generator computer program
Randomization unit
Authors: One of the three edited abstracts will be randomly selected for bonus payments. Reviewers: One of the twenty comparison pairs will be randomly selected for bonus payments.
Clusters
Authors: 300 participants Reviewers: 140 participants recruited so far
Observations
Authors: Each participant submits decisions for 3 abstracts. Reviewers: Each participant submits decisions for 20 pairs of original and edited abstracts
Arms
Authors: Four treatments and 75 participants per treatment. No treatment arms for the Reviewers
Primary outcomes
Author: number of edits, type of edits Reviewers: decisions about the edited abstract in reference to the original abstract

Cited by

No citation data available for this registration yet.

Keywords