Accepted with Revisions: Evaluating AI-Assisted Scientific Writing
How to cite
Hazra, Sanchaita; Lee, Doeun; Majumder, Bodhisattwa Prasad; Kumar, Sachin. 2025. "Accepted with Revisions: Evaluating AI-Assisted Scientific Writing." AEA RCT Registry. AEARCTR-0016740. https://doi.org/10.1257/rct.16740-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Large Language Models (LLMs) have seen expanding application across domains, yet their effectiveness as assistive tools for scientific writing—an endeavor requiring precision, multimodal synthesis, and domain expertise—remains insufficiently evaluated. We examine the potential of LLMs to support domain experts in scientific writing, with a focus on abstract composition. We design an incentivized randomised trial with a hypothetical conference set up where participants with relevant expertise are segregated into authors and reviewers. Inspired by methods in behavioral science, our novel incentive structure encourages participants to produce high-quality outputs. Authors edit original (control) or their AI-generated (treatment) abstracts of published research from top-tier conferences. Reviewers evaluate if the edited abstract provides adequate justice to the research presented in the original abstract.
Intervention
We simulate a realistic writing environment where authors (domain experts), the intended users of an AI assistant, use and edit scientific abstracts to create scientific prose to get it accepted by atleast two reviewers. We have a 2x2 design for the Author study. The first dimension includes the source of the abstract: original or AI-generated. The second dimension includes information disclosure or not about the source of the abstracts. No treatments for Reviewers who review the edited abstract in reference to the original abstract and answer if the edited abstract provides adequate justice to the research idea presented in the original abstract.
Experimental design
We simulate a realistic writing environment where authors (domain experts), the intended users of an AI assistant, use and edit scientific abstracts to create scientific prose to get it accepted by atleast two reviewers. We have a 2x2 design for the Author study. The first dimension includes the source of the abstract: original or AI-generated. The second dimension includes information disclosure or not about the source of the abstracts. No treatments for Reviewers who review the edited abstract in reference to the original abstract and answer if the edited abstract provides adequate justice to the research idea presented in the original abstract. We collect secondary beliefs like confidence and readability of abstracts.
Design details
Cited by
No citation data available for this registration yet.