Evaluating an AI-Powered Research Development Tool for Academic Productivity and Well-being

How to cite

Polanco-Jimenez, Jaime; Sevilla, Almudena. 2026. "Evaluating an AI-Powered Research Development Tool for Academic Productivity and Well-being." AEA RCT Registry. AEARCTR-0017749. https://doi.org/10.1257/rct.17749-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jaime Polanco-Jimenez* , Almudena Sevilla

* Primary investigator

Registry ID
AEARCTR-0017749
DOI
https://doi.org/10.1257/rct.17749-2.0
First registered
Last updated
Status
In development
Trial dates
2026-02-01 – 2028-06-30
Countries
Private
JEL codes
A23, C93, J24, O31, L86, I31
Registry record
http://www.socialscienceregistry.org/trials/17749

Abstract

This randomized controlled trial (RCT) evaluates the causal impact of an AI-powered Research Development Tool on the academic productivity and well-being of early-career researchers. Participants — PhD students and junior economists within five years of their doctorate — will be randomly assigned, within career-stage strata, to either a control group with access to a general-purpose AI or a treatment group with access to a comprehensive AI-driven Research Development Suite offering structured, expert-level feedback on research papers. The control group will receive access 12 months after the experiment's start date. Over a 24-month intervention period, we measure changes in externally evaluated research quality, paper submission rates, job satisfaction, and work-life balance.
Our central question is whether the productivity effects of AI access vary systematically with the researcher's level of accumulated tacit knowledge and structural advantage — specifically evaluating heterogeneity by career stage and controlling for baseline institutional prestige. We test whether AI acts as a substitute for elite networks (compressing inequality) or a complement to existing advantages (amplifying inequality).

Intervention

This is a two-arm, parallel-group, single-blind randomized controlled trial (RCT). Participants — PhD students and junior economists within five years of their doctorate — will be randomly allocated to either a control group or a treatment group in a 1:1 ratio, using stratified randomization by career stage. The intervention will last for 24 months. Data will be collected through baseline and endline surveys and continuous behavioral logging from the AI platform. The study aims to evaluate whether the productivity effects of AI access vary systematically with the researcher's level of accumulated tacit knowledge.

Experimental design

This is a two-arm, parallel-group, single-blind randomized controlled trial (RCT) evaluated over a 24-month period. The study investigates the causal impact of a structured, AI-powered Research Development Tool on the academic productivity of early-career researchers. Participants will be stratified by career stage (PhD students vs. junior economists within five years of their doctorate) and randomized in a 1:1 ratio into either a treatment group or a waitlist control group.

The treatment group will receive immediate access to the AI tool, which provides expert-level feedback on working papers. The waitlist control group will conduct their research as usual (which may include ad-hoc use of general-purpose LLMs) and will receive access to the treatment tool only after the 12-month intervention concludes. Our primary objective is to measure changes in real-world publication milestones (e.g., desk rejection rates, Revise & Resubmit invitations, and working paper circulation). Secondarily, we will test whether the treatment effect varies systematically by the researcher's level of accumulated tacit knowledge (career stage), identifying whether AI access compresses or amplifies existing inequalities in research output.

Design details

Randomization method
Individual participants will be randomly assigned to one of the two arms using a computerized stratified random assignment. Stratification is based on one binary variable: Career Stage (PhD Student vs. Junior Economist).
Randomization unit
Individual participant
Clusters
Not applicable (individual-level randomization).
Observations
Our primary recruitment target is 504 participants (252 per arm), which provides 80% power to detect a moderate effect (d = 0.50) on the main ATE and 80% between treatment and career stage. This is our reference scenario given that existing literature on AI and cognitively demanding tasks (Brynjolfsson et al., 2023; Noy & Zhang, 2023) reports effects in the small-to-medium range. Gender interactions are pre-specified as exploratory: detecting a triple interaction (treatment × career stage × gender) requires approximately four times the sample needed for the main interaction.
Arms
Arm 1 (Control Group - waitlist/status quo): 252 participants Arm 2 (Treatment Group - RefereeAI Suite): 252 participants
Primary outcomes
1. Change in objective academic progression milestones over the 24-month period, specifically measured by: (a) rate of desk rejections, (b) invitations to Revise and Resubmit (R&R), and (c) total number of working papers publicly circulated (e.g., NBER, CEPR, SSRN, or university working paper series) 2. Heterogeneity of the treatment effect on objective publication milestones by career stage: whether the impact of AI access on reducing desk rejections and increasing R&Rs, and the amount of research, differs systematically between PhD students and junior economists. 3. Change in the number of papers submitted for publication. 4. Change in self-reported job satisfaction. 5. Change in self-reported work-life balance satisfaction.
Secondary outcomes
1. Change in algorithmic paper quality (0-100 score). At the endline, both treatment and control group working papers will be processed blindly through the 'Editor Agent' prompt of the ResearchAI tool to calculate a standardized quality score. This serves as a mechanistic measure of algorithmic compliance rather than a measure of ultimate scientific validity.

Cited by

No citation data available for this registration yet.

Keywords