Causal Impact of Content-Aligned Expertise on Grant Peer-Review Predictive Performance

How to cite

Franzoni, Chiara. 2025. "Causal Impact of Content-Aligned Expertise on Grant Peer-Review Predictive Performance." AEA RCT Registry. AEARCTR-0017412. https://doi.org/10.1257/rct.17412-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Chiara Franzoni*

* Primary investigator

Registry ID
AEARCTR-0017412
DOI
https://doi.org/10.1257/rct.17412-1.0
First registered
Last updated
Status
In development
Trial dates
2025-12-09 – 2026-02-28
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/17412

Abstract

This experimental study uses randomized matching to identify the causal effect of reviewer content expertise on the predictive validity of grant-proposal evaluations. The principle of “deference to expertise” assumes that reviewers whose expertise is more closely aligned with a proposal’s content should be more likely to predict the expected social value of proposals than reviewers with weaker content alignment. Identifying the causal impact of expertise alignment with observational data is challenging, because expertise is the primary criterion for review assignment generating endogeneity in content-alignment of assignments. We overcome this challenge by experimentally randomizing reviewers into high- or low-alignment conditions, enabling identification of the causal impact of content expertise on predictive performance.

Intervention

This experimental study uses random assignment of proposals to reviewers to identify the causal effect of reviewer content alignment on predictions evaluations. For each reviewer–proposal pair, we compute an ex-ante measure of content alignment between each reviewers’ prior publications and the text of the proposals. Random assignment of proposals to reviewers therefore induces exogenous variation in reviewer content expertise across evaluations. Because expertise measures are constructed entirely from pre-treatment information, observed differences in outcomes across levels of expertise can be causally attributed to this variation.

Experimental design

We pre-screen participants to check the alignment between their background expertise and each proposal content. We use this information to identify eligible reviewers as those who are minimally competent and have at least one high and one low content alignment match. We then randomly assign them to one of two conditions: High-alignment (they evaluate a proposal selected at random from their high-alignment options); Low-alignment (they evaluate a proposal selected at random from their low-alignment options).

After receiving their assigned proposal, reviewers provide predictions on the project’s future research impact. The causal effect of content alignment on prediction accuracy is estimated by comparing prediction accuracy across the two randomly assigned groups, while accounting for proposal-level differences.

Design details

Randomization method
Randomization is done in office, using a random generator provided by a common software (e.g., Excel or Python).
Randomization unit
Randomization units are individual-proposal matches. Each individual-proposal match can result in a low-content alignment match or high- content alignment match.
Clusters
Minimum 490 observations. The ideal sample we hope to achieve is about twice as big.
Observations
Minimum 490 observations. We hope to achieve a sample about twice as big. However, the final number of observations is difficult to predict, as it may be substantially affected by attrition, given that we need to recruit very qualified participants (scientists who publish in the biomedical literature). Even if these participants have signed-up to volunteer in the study, it is customary to in academia to turn-down peer review jobs for time-constraints.
Arms
We expect the proportion in treatment to be between 0.4-0.5, and thus aim for a minimum of 230 in each treatment arm.
Primary outcomes
Prediction errors
Secondary outcomes
Aggregation benefit (Wisdom-of-the-crowd)

Cited by

No citation data available for this registration yet.

Keywords