Paper quality

How to cite

Feld, Jan. 2020. "Paper quality." AEA RCT Registry. AEARCTR-0005620. https://doi.org/10.1257/rct.5620-3.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jan Feld*

* Primary investigator

Registry ID
AEARCTR-0005620
DOI
https://doi.org/10.1257/rct.5620-3.0
First registered
Last updated
Status
Completed
Trial dates
2020-04-28 – 2021-12-31
Countries
Australia (), New Zealand ()
Registry record
http://www.socialscienceregistry.org/trials/5620

Abstract

This project is about perceived quality of papers.

Experimental design

Nothing to report before the experiment is completed.

Design details

Randomization method
Randomization for main analysis sample: We split the papers by topic into a micro-economics group (20 papers) and a macro-economic group (10 papers). We randomly assign each paper within a group one “paper number”, so that papers in the micro group will be assigned numbers 1-20 and papers in the macro group will be assigned numbers 21-30. After that, we create the following six paper blocks: 1. Papers 1-10 (micro), papers 1-5 original version, papers 6-10 edited version 2. Papers 1-10 (micro), papers 1-5 edited version, papers 6-10 original version 3. Papers 11-20 (micro), papers 11-15 original version, papers 16-20 edited version 4. Papers 11-20 (micro), papers 11-15 edited version, papers 16-20 original version 5. Papers 21-30 (macro), papers 21-25 original version, papers 26-30 edited version 6. Papers 21-30 (macro), papers 21-25 edited version, papers 26-30 original version Randomization for our language expert sample: We randomly assign each paper one “language paper number”. This is a different randomization, the “language paper numbers” are not the same as the “paper numbers”. After that, we create the following six blocks: 1. Papers 1-10, papers 1-5 original version, papers 6-10 edited version 2. Papers 1-10, papers 1-5 edited version, papers 6-10 original version 3. Papers 11-20, papers 11-15 original version, papers 16-20 edited version 4. Papers 11-20, papers 11-15 edited version, papers 16-20 original version 5. Papers 21-30, papers 21-25 original version, papers 26-30 edited version 6. Papers 21-30, papers 21-25 edited version, papers 26-30 original version This assignment of papers to paper numbers will be done with a computer. The papers will be presented in random order to each rater. This randomization will be done within the survey by Qualtrics. Assignment of raters to paper blocks For economists: We create two lists of academic economists from Australian universities or research institutes: one for micro economists and one for macro economists. We invite economists in both groups via email to evaluate 10 academic papers in their discipline. Those who agree to evaluate ten papers are assigned to a paper block, and we will send them a survey containing links to the relevant papers. More specifically, micro-economists will be assigned to paper blocks 1-4 (which contain micro papers) and macro-economists will be assigned to paper blocks 5-6 (which contain macro papers). The order of the paper block assignment will be determined by the order the economists agree to participate. For example, the first micro-economist will be assigned to paper block 1, the second micro-economist will be assigned to paper block 2, etc. Similarly, the first macro-economist will be assigned to paper block 5 and the second macro-economist will be assigned to paper block 6. We will deviate from this assignment-procedure to avoid that economists from the same institution are asked to judge different versions of the same paper within a short time. Economists from the same institution are more likely to talk to each other about the task and therefore might realize that we have included different versions of the same paper in the experiment. This may raise suspicions, which we want to avoid. We will solve this problem by swapping the assignment to paper blocks with economists from other institution. For example, if two economists from the University of Melbourne would have been assigned to paper-blocks 1 and 2, and one economist from the University of Sydney would have been assigned to paper-block 3, we would swap the paper-block assignments of the second University of Melbourne economist with the University of Sydney economist. For language experts: We will invite language experts to evaluate 10 academic papers. If they agree, we will send them a survey containing links to the relevant papers. The order of the paper block assignment will be determined by the order the language experts agree to participate. The expert who first agrees to participate will be assigned to paper block 1, the second expert will be assigned to paper block 2, etc. As for the academic economists, we will deviate from this assignment procedure to avoid that experts from the same institution are asked to judge different versions of the same paper.
Randomization unit
See explanation above. Each paper will be randomly assigned to a paper number (randomization unit = paper) Each rater will be randomly assigned to a rater number (randomization unit = rater)
Clusters
30 clusters (one cluster for each paper).
Observations
For our main analysis, we expect to have 300 paper-version-rater observations. For the analysis by language experts, we expect to have 180 paper-version-rater observations.
Arms
Main analysis: Treatment group: 30 edited paper versions (= 150 paper-version-rater observations) Control group: 30 non-edited paper versions (= 150 paper-version-rater observations) Language expert analysis: Treatment group: 30 edited paper versions (= 90 paper-version-rater observations) Control group: 30 non-edited paper versions (= 90 paper-version-rater observations)
Primary outcomes
1. Estimated effect of language editing on writing quality. 2. Estimated effect of language editing on paper quality. 3. If we find significant effects for primary outcome 1, we will also estimate the effect of writing quality on paper quality.
Secondary outcomes
We have three secondary outcomes, which show the effect of language editing on economists’ assessment of the following. 1. The chance of getting the paper accepted at a conference. 2. Conditional on submission, the chance of the paper getting desk rejected. 3. The probability of the paper getting published in a good journal. If we find significant effects of paper language editing on writing quality, we will also estimate the effect of writing quality on economists’ assessment of the following. 4. The probability of getting the paper accepted at a conference. 5. The probability of getting desk rejected. 6. The probability of getting published in a good journal. We have five secondary outcomes which are based on the evaluations of language experts. These show the effect of language editing on: 7. The writing quality. 8. How easy the paper allows the reader to find the key messages. 9. To what extent the paper is free of spelling and grammar mistakes. 10. How easy to read the paper is. 11. How concise the paper is.

Materials

Papers from this trial

  1. Feld, J., Lines, C., & Ross, L. (2023). Writing matters. https://www.iza.org/publications/dp/16571

Cited by

No citation data available for this registration yet.

Keywords