Affiliation bias and conference inclusion: Experimental evidence from early career researchers
How to cite
Favero, Luca; Alonso, Agueda Solis; Carreras, Enrique; Gallegati, Giacomo. 2023. "Affiliation bias and conference inclusion: Experimental evidence from early career researchers." AEA RCT Registry. AEARCTR-0012229. https://doi.org/10.1257/rct.12229-1.3
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We study the extent author affiliation can bias reviewers in grading papers submitted to international conferences. We exploit a PhD Workshop in Economics arranged by local PhD candidates. Our objective is to investigate the extent to which author affiliation can introduce bias in the evaluation of research papers. Affiliation is often used as a signal to form beliefs about research quality (Blank 1991). We test whether signals can potentially be misinterpreted and exacerbate inequalities across researchers from differently ranked institutions. We will compare blind and non-blind assessments for each paper submitted to the conference.
Intervention
We study the extent author affiliation can bias reviewers in grading papers submitted to international conferences. Affiliation is often considered as a signal for quality in research outputs. Nevertheless, lack of experience with research, the review process or field knowledge can lead to over reliance on affiliation. This can exacerbate inequalities between researchers from high ranked institutions and those affiliated to less prestigious establishments.
We randomly allocate reviewers to either blind or not blind paper grading. Within each treatment arm we will randomly allocate papers to reviewers. We remove from all papers information on title, authorship and acknowledgements. Papers assigned for blind grading also have authors’ affiliation removed, while papers not assigned for blind grading keep the affiliation of the submitter visible. We only retain the submitter’s affiliation(s) for co-authored papers. Affiliation(s) will be positioned in a footnote in the title page. Reviewers will be instructed not to speak to each other about the reviews. We will not disclose to reviewers that an experiment is taking place at any time. Reviewers will be asked not to communicate to avoid coordinating on grading. Reviewers will not have information on their peers’ grading. Papers are distributed to reviewers through a dedicated shared folder that is only accessible to them. We will rename blind and non-blind papers following separate naming conventions to increase coordination costs for reviewers. In addition to this, both blind and non-blind papers will be made available as non-searchable PDF documents to increase the cost of searching for the missing information online.
All papers are evaluated by two reviewers, one assigned to blind grading and one assigned to non-blind grading. Reviewers are asked to assess papers along three grading criteria: Grade A is assigned based on the research question developed in the paper. Grade B refers to writing style while Grade C assess the research design. All grades range from 1 to 10. Reviewers will also be asked a recommendation on acceptance. For each paper we will also include an optional open-ended box for reviewers to share any comment they wish to provide. The same data collection tool will include questions on beliefs on the relative quality of their own work compared to the submitted papers and willingness to meet the author of the paper.
Subsequently to collecting grades, we will ask reviewers a number of questions to assess any threats to validity. We will inquire reviewers’ about their perceived purpose of this grading scheme. Next, we will ask reviewers to rank universities into separate tiers based on their perceived quality of institutions. Finally, we will also collect information on perceived importance for title, name, affiliation and acknowledgments in evaluating papers.
Reviewers will receive a grading package up to 3 days after the submission deadline to allow for paper allocation and anonymization. The grading package consists of instructions for grading and either blind or not blind paper versions based on a reviewer randomized assignment. Reviewers are instructed not to share or talk to each other about the grading package they receive. Reviewers have up to 14 days to grade about 10 papers each. Grades are sent back to the conference organizers via a dedicated excel tool. After obtaining grading data and perceived purpose from all reviewers, we will collect information on beliefs about relative quality, university rankings and the perceived importance of different signals through a dedicated survey.
Experimental design
We compare reviews for the same paper by blind and non-blind reviewers. We will use the whole body of submissions to a PhD Conference in Economics organized in Italy. Submissions are open to PhD students from any field of Economics and all universities. Local PhD candidates will review submissions. Reviewers are first stratified by research area (either micro theory, applied micro and macro) and then randomly allocated to either blind or non-blind grading. Papers are then randomly assigned within each group according to the research area of the paper. Every paper is read by one blind reviewer and a non-blind reviewer. Reviewers submit their grades within 14 days after receiving the grading package.
We explore grading differences between blind and non-blind assessments on quality of the research idea, soundness of the research design and writing style. Affiliation bias is defined as any difference, for each paper, between blind and non-blind assessments. We will then assess whether affiliation bias plays any role in changing beliefs about quality of own work. We will explore mechanisms by assessing heterogeneous treatment effects for universities positioned at the top and bottom of the underlying quality distribution. We will also explore any qualitative insights from open ended text shared by reviewers and explore networking as a potential mechanism.
Finally, we will also investigate whether seniority in the PhD program and mismatch between reviewers and papers’ topic play a role in shaping affiliation bias.
Design details
Cited by
No citation data available for this registration yet.