Competition and Quality in Science
How to cite
Stein, Carolyn; Hill, Ryan. 2023. "Competition and Quality in Science." AEA RCT Registry. AEARCTR-0011356. https://doi.org/10.1257/rct.11356-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This is a survey experiment to assess whether competition affects the quality of scientific research. We will reach out to corresponding authors on scientific papers via a fully anonymous email survey.
We will cross-randomize across two sets of questions. The first set of questions will assess whether scientists think that important work is more likely to be competitive. We will pose a hypothetical scenario about working on a project, but randomly vary the expected journal placement of the project. We will then ask scientists how likely they think it is that another team is working on a similar project.
The second set of questions will assess whether scientists work faster and less carefully if they believe the project is competitive (i.e., multiple parties are working on the same project). We will pose a hypothetical scenario about working on a project, but randomly vary the probability that another team is working on the project. We will then ask scientists (a) how long will they spend finishing the project and (b) which tasks would the do prior to publication (replication, additional analyses, etc.)
Intervention
The intervention is an optional email survey conducted via Qualtrics. No PII will be collected. See the experimental design for more details.
Experimental design
STUDY POPULATION:
We have selected 11 scientific fields, using field classifications from Microsoft Academic Graph (MAG). MAG assigns publications to fields. Researchers were assigned to fields using the modal field of their publications. We had two criteria when selecting the fields. (1) The field should be sufficiently large such that we have a large enough sample; (2) the field should be largely experimental. We proxy for experimental by flagging whether the title or abstract contains the word "experiment." The selected fields are the following:
Structural biology
Cell biology
Ecology
Horticulture & agronomy
Immunology
Biochemistry
Inorganic chemistry
Condensed matter physics
Optics
Social psychology
SURVEY EXPERIMENT:
Every recipient will be sent an email survey via Qualtrics. Two of the question modules will be cross-randomized.
Module 1. Randomized text is in the square brackets.
Consider the following scenario: You are working on a project and you have generated some preliminary results. Based on the research question and your results, you expect that it will publish in a [high impact journal (such as Science, Nature, or the top journal in your field) / medium impact field journal].
Q1. How likely is it that another research team is working on a very similar project? (0 to 100% slider)
Module 2. Randomized text is in the square brackets.
Consider a different scenario: Suppose you have generated some preliminary results for a project. You are fairly confident that [nobody else is working on a very similar project (less than a 10% chance) / another team is working on a very similar project (greater than a 90% chance)]. Answer the following three questions with this scenario in mind.
Q1. How long would it take for you to complete the project and submit the paper to a journal? (sliding scale 0 to 24 months)
Q2. Which of the following would you do prior to submitting the paper?
Re-run or replicate key experiments (Yes/Maybe/No/NA)
Run additional supporting experiments (Yes/Maybe/No/NA)
Perform a thorough code review (Yes/Maybe/No/NA)
Perform a thorough review of any mathematical or analytical analyses (Yes/Maybe/No/NA)
Perform a thorough proofreading of the manuscript (Yes/Maybe/No/NA)
Q3. Would you do anything else prior to submission not listed above? (free-form text; optional)
Module 3. No randomization
Q1. In general, how would you rate the competition to publish first in your field? (1-4 scale)
Q2. In general, do you feel that peers in your field ever sacrifice the quality of their research in order to publish first? (1-4 scale)
Q3. When you are reading published research in your field, how do you assess the quality of the work? Is there a systematic way to measure high quality, careful research in your field? (free form text; optional)
Design details
Cited by
No citation data available for this registration yet.