The Null Result Penalty

How to cite

Roth, Christopher; Haaland, Ingar; Chopra, Felix; Stegmann, Andreas. 2022. "The Null Result Penalty." AEA RCT Registry. AEARCTR-0009513. https://doi.org/10.1257/rct.9513-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Christopher Roth* , Ingar Haaland , Felix Chopra , Andreas Stegmann

* Primary investigator

Registry ID
AEARCTR-0009513
DOI
https://doi.org/10.1257/rct.9513-1.0
First registered
Last updated
Status
Completed
Trial dates
2022-04-26 – 2022-05-12
Countries
United States of America ()
Registry record
http://www.socialscienceregistry.org/trials/9513

Abstract

We examine how the evaluation of research studies in economics depends on
whether a study yielded a null result. Studies with null results are perceived to
be less publishable, of lower quality, less important, and less precisely estimated
than studies with significant results, even when holding constant all other study
features, including the precision of estimates. The null result penalty is of similar
magnitude among PhD students and journal editors. The penalty is larger when
experts predict a large effect and when statistical uncertainty is communicated
with p-values rather than standard errors. Our findings highlight the value of
pre-results review.

Experimental design

See the paper.

Design details

Randomization method
Computer
Randomization unit
Respondent
Clusters
480 in Experiment 1 and 95 in Experiment 2.
Observations
575
Arms
287 null treatment 288 non-null treatment
Primary outcomes
Perceptions of publishability, quality, importance and precision.

Cited by

No citation data available for this registration yet.

Keywords