Bias in Peer Review

How to cite

Levine, S; Williams, H; Stein, C. 2022. "Bias in Peer Review." AEA RCT Registry. AEARCTR-0008755. https://doi.org/10.1257/rct.8755-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

S Levine* , H Williams , C Stein

* Primary investigator

Registry ID
AEARCTR-0008755
DOI
https://doi.org/10.1257/rct.8755-1.0
First registered
Last updated
Status
In development
Trial dates
2022-02-15 – 2035-12-31
Countries
Private
JEL codes
J14 L38 M14 O31
Registry record
http://www.socialscienceregistry.org/trials/8755

Abstract

In many scientific contexts, peer review can be either single-blind or double-blind: in single-blind review, research work (e.g., manuscripts, proposals) is reviewed alongside information on the author(s), whereas in double-blind review, information on the author(s) is withheld. We will report results from a randomized experiment conducted in collaboration with a grantmaking body, in the context of the grantmaker reviewing proposals in one field of science. By comparing scores across single-blind and double-blind reviews for groups of different characteristics, we can test for potential biases in the review process.

Intervention

Research proposals received by the grantmaking will be randomly assigned to reviewers (while avoiding conflicts of interest, as defined by the grantmaking body). Then, each assigned reviewer–application pair will be further randomly assigned to either:

a. Single-blind review, in which the reviewer sees both the project description and the applicant's information
b. Double-blind review, in which the reviewer sees only the project description but not the applicant's information
c. Project-blind review, in which the reviewer sees only the applicant's information but not the project description

Experimental design

The experiment empirically tests for bias in the grantmaking process

Design details

Randomization method
We wrote a program in the statistical language R, and representative of the grantmaking body executed it on their computers. The program took three inputs: 1. The list of reviewers 2. The list of applications 3. Any reviewer-application pairs that represented a conflict of interest. As described above, there are three types of reviews: single-blind, double-blind, and project-blind. Starting with single-blind reviews, randomly assign applications to reviewers. If a reviewer receives an application that represents a conflict of interest or an application they have previously been assigned, that application is returned to the pool, and a new application is drawn. We repeat this procedure of double-blind and project blind applications.
Randomization unit
Individual application in a given year
Clusters
The sample size will be the number of applications received by the grantmaking body.
Observations
The sample size will be the number of applications received by the grantmaking body.
Arms
The sample size will be determined by the number of applicants to the program over the life of this experiment. For instance, in 2022, the grantmaking body received exactly 100 applications. This implies 400 single-blind reviews (4*100), 200 double-blind reviews (2*100), and 200 project-blind reviews (2*100).
Primary outcomes
Our key outcome variable of interest is the score assigned to an application by a reviewer.

Cited by

No citation data available for this registration yet.

Keywords