Causal Impact of Content-Aligned Expertise on Grant Peer-Review Predictive Performance
How to cite
Franzoni, Chiara. 2025. "Causal Impact of Content-Aligned Expertise on Grant Peer-Review Predictive Performance." AEA RCT Registry. AEARCTR-0017412. https://doi.org/10.1257/rct.17412-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This experimental study uses randomized matching to identify the causal effect of reviewer content expertise on the predictive validity of grant-proposal evaluations. The principle of “deference to expertise” assumes that reviewers whose expertise is more closely aligned with a proposal’s content should be more likely to predict the expected social value of proposals than reviewers with weaker content alignment. Identifying the causal impact of expertise alignment with observational data is challenging, because expertise is the primary criterion for review assignment generating endogeneity in content-alignment of assignments. We overcome this challenge by experimentally randomizing reviewers into high- or low-alignment conditions, enabling identification of the causal impact of content expertise on predictive performance.
Intervention
This experimental study uses random assignment of proposals to reviewers to identify the causal effect of reviewer content alignment on predictions evaluations. For each reviewer–proposal pair, we compute an ex-ante measure of content alignment between each reviewers’ prior publications and the text of the proposals. Random assignment of proposals to reviewers therefore induces exogenous variation in reviewer content expertise across evaluations. Because expertise measures are constructed entirely from pre-treatment information, observed differences in outcomes across levels of expertise can be causally attributed to this variation.
Experimental design
We pre-screen participants to check the alignment between their background expertise and each proposal content. We use this information to identify eligible reviewers as those who are minimally competent and have at least one high and one low content alignment match. We then randomly assign them to one of two conditions: High-alignment (they evaluate a proposal selected at random from their high-alignment options); Low-alignment (they evaluate a proposal selected at random from their low-alignment options).
After receiving their assigned proposal, reviewers provide predictions on the project’s future research impact. The causal effect of content alignment on prediction accuracy is estimated by comparing prediction accuracy across the two randomly assigned groups, while accounting for proposal-level differences.
Design details
Cited by
No citation data available for this registration yet.