Reproducibility in research – Ensuring the transparency and credibility of your work
2025-05-26
Follow along
larsvilhuber.github.io/summer-school-qicss-2025/presentation/presentation.html (PDF)
Who are we?
Your instructors
Michael Stepner
Assistant professor at the University of Toronto and a Research Principal at Opportunity Insights, a lab based at Harvard University.
Lars Vilhuber
Executive Director of the Labor Dynamics Institute and Senior Research Associate in the Economics Department at Cornell University, and the American Economic Association’s Data Editor.
Data Editor of the AEA
2389 Manuscripts and 4440 Reports, approx. 4400 authors reached.
Some housekeeping
Languages
- All slides are in English
- Primary language is English
- Toutes les questions peuvent être en français
- Toutes les réponses vont être dans la langue posée, ou en anglais
Code of Conduct
We are dedicated to providing a welcoming and supportive environment for all people, regardless of background or identity. By participating in this team, participants accept to abide by LDI ReplicationLab’s Code of Conduct and accept the procedures by which any Code of Conduct incidents are resolved. Any form of behaviour to exclude, intimidate, or cause discomfort is a violation of the Code of Conduct. In order to foster a positive and professional learning environment we encourage the following kinds of behaviours in all platforms and events:
Code of Conduct
- Use welcoming and inclusive language
- Be respectful of different viewpoints and experiences
- Gracefully accept constructive criticism
- Show courtesy and respect towards other community members
If you believe someone is violating the Code of Conduct, we ask that you report it to us, QICSS/CIQSS (sponsor), or the CEA (host).
Walkthrough of the agenda
Today
- 13:00 Welcome
- 13:05 Walkthrough
- 13:15 Goals
- 13:30 Technical setup, possible team formation
- 13:45 🔒Hands-on Exercise: A very imperfect example
- 14:00 Motivation: A tale of ineffective technical collaboration
- 14:05 Day 1: Setting yourself up for reproducibility
- 16:00 End of day one
Tomorrow
- 9:00 Discussion of the “Very imperfect example”
- 9:30 Documenting it all: How to correctly document a replication package (and why!)
- 10:30 Break
- 10:45 How to run Stata! or R! (reproducibly)
- 11:00 Extra: How to install Stata packages
- 11:15 Topic A (see Survey)
- 12:00 Break
- 13:00 🔒When data cleaning is 🔒critical
- 13:30 Topic B (see Survey)
- 14:15 Break
- 14:30 Hands-on: Improving the replication package (very imperfect -> a lot better)
- 15:00 Hands-on: Testing it all
- 15:15 Wrap up
- 16:00 Fin.
Best practices?
First: why?
Why reproducibility?
- Credibility
- Transparency (openness)
- Efficiency of scholarly discourse?
Why reproducibility?
- Early publications (20th century) contained tables of data, and the math was simple (maybe)
- Data became electronic, were no longer included or cited
- Math was transcribed to code, and was no longer included
Increasing broad consensus in academia
- FAIR principles
- Data Citation Principles
- Computational Reproducibility
- Findable
- Accessible
- Interoperable
- Reusable
Data Citation Principles
- FAIR principles
- Data Citation Principles
- Computational Reproducibility
- To make it findable, citations,
- Give attribution and credit for data.
Computational reproducibility
- FAIR principles
- Data Citation Principles
- Computational Reproducibility
- Primary topic today
Reproducibility means obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis.2
What is…
What is a replication package?
A Replication Package is
- Code
- Data
- Materials (for surveys, experiments, …)
- Instructions on how to obtain data not included
- Instructions on how to combine it all
- Known issues documented
Complies with…
Is stored in…
- AEA Data and Code Repository
- Other trusted repositories
Best practices?
Summing up
- Why
- Credibility
- Transparency (openness)
- Efficiency of scholarly discourse ([example])
- How
- FAIR principles
- Data Citation Principles
- Computational Reproducibility
- As Replication Packages
- Code
- Data
- Materials (for surveys, experiments, …)
- Instructions on how to obtain data not included
- Instructions on how to combine it all
- Known issues documented
Who?
Who?
- 🐇 Authors at conditional acceptance
- 🐢 Authors at submission
- 🐁 Authors at beginning of project
- 👴🏻👵🏽 Experienced researchers
- 👶🏽👶🏻 Junior researchers
- 👨🎓👩🎓 Ph.D. students
- 🧒👦 Undergraduates
Who?
You.
You
👶🏻 Now:
- more efficient development
- more efficient collaboration
- more assurance that “everything just works”
👵🏽 Soon
- more efficient development across projects
- more efficient response to editors and referrees
- … while you are in a new institution, on a new computer, with three courses to prep, and (luxury!) a RA you can delegate to…
You
How?
How to create reproducible research?
Habits
- Reproducibility from Day 1
- Adopt reproducible habits
- Take notes when you do things, not after
- Use version control
Strategy
Computational empathy: think of the next person to run this - It could be you in 5 years!
Hands-on: A very imperfect example
Day 1 reproducibility
Day 1: How to not to organize your work
Day 1: Setting yourself up for reproducibility
End of Day 1
Appendix
Where to?
Choices
- Issue No. 1: Data Access and preservation 💥 [a]
- Issue No. 2: Confidential data - same ⬆️! [a] [b]
- Lifecycle checking: Self-checking reproducibility and presentation
- New challenges: AI and Big Data but same ⬆️!
- New methods: Transparency outsourced or certified
- Implementation in academia: Students!
Resources
README
Lars Vilhuber, Connolly, M., Koren, M., Llull, J., & Morrow, P. (2022). A template README for social science replication packages (v1.1). Social Science Data Editors. https://doi.org/10.5281/zenodo.7293838
You can download the Word, LaTeX, or Markdown version of the README with lots of examples.
Other guidance
Presentation on “Self Checking Reproducibility” and its associated website
Guidance when (some) data are confidential: https://labordynamicsinstitute.github.io/reproducibility-confidential/
Guidance for citations: https://social-science-data-editors.github.io/guidance/addtl-data-citation-guidance.html
Extra info
- This document’s source: https://github.com/larsvilhuber/summer-school-qicss-2025
- Licensed under
![CC BY-NC 4.0]()
Sources
- Images: NYT, Bluesky 1, 2, Ike Hayman/Wikimedia
Details on Transparency, etc.
Transparency
- Provenance of the data
- Processing of the data, from raw data to results (code)
It is the policy of the American Economic Association to publish papers only if the data used in the analysis are clearly and precisely documented and access to the data and code is clearly and precisely documented and is non-exclusive to the authors.
Completeness
- All data needs to be identified and and access described
- All code needs to be described and provided
- All materials must be provided (survey forms, etc.)
Authors … must provide, prior to acceptance, the data, programs, and other details of the computations sufficient to permit replication
Preservation
- All data needs to be preserved for future replicators
- Ideally, within the replication package, subject to ToU, for convenience
- Otherwise, in a trusted repository
Preservation
- Code must be in a trusted repository
- Usually, within the replication package
- Websites, Github, are not acceptable
Historically
AER 1911 thanks to Stefano Dellavigna
Modern preservation
Exceptions to the Policy
None
…
… there is a grey zone:
- When data do not belong to researcher, no control over preservation, access!
- Sometimes, ToU prevent researcher from revealing metadata (name of company, location)
Transparency again
- However:
- No exception for need to describe access (own and other)
- No exception for need to fully describe processing (possibly with redacted code)
Reproducibility in Economics and beyond
Data Editors
Common policies
https://social-science-data-editors.github.io/
Elsewhere: Political Science
Elsewhere: Sociology
But!
Elsewhere: Sociology
.jpg){kind=link}
Benefits
Building on the work of others
Roth, Jonathan. 2022. “Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends.” American Economic Review: Insights 4 (3): 305–22. DOI: 10.1257/aeri.20210236
Notes: “I exclude 43 papers for which data to replicate the main event-study plot were unavailable.”
Building on the work of others: dCdH 2020
de Chaisemartin, Clément, and Xavier D’Haultfœuille. 2020. “Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects.” American Economic Review 110 (9): 2964–96. DOI: 10.1257/aer.20181169
The results from various other papers are recomputed to empirically demonstrate the relevance of the proposed methods.
Transparency elsewhere
Transparency outsourced
Transparency outsourced
- A third party conducts the reproducibility, not you, not me.
- Need to common understanding, protocols, etc.
- AEA’s protocol
- We do this about a dozen times per year
Transparency outsourced
Why should I believe the third party?
- Trust
- Transparency
- Common methods
Transparency certified
Transparency certified
- Providing information about the computing platforms themselves, including specific details about how computational transparency is supported.
- Packaging and signing resulting artifacts along with records of their execution using a standard format.
Applications
- Limor, R-squared, cascad, World Bank!
- FSRDC? IRS?
- Meta data?
Footnotes
Data Citation Synthesis Group: Joint Declaration of Data Citation Principles. Martone M. (ed.) San Diego CA: FORCE11; 2014 https://www.force11.org/group/joint-declaration-data-citation-principles-final
National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303.
Weeden, K. A. (2023). Crisis? What Crisis? Sociology’s Slow Progress Toward Scientific Transparency . Harvard Data Science Review, 5(4). https://doi.org/10.1162/99608f92.151c41e3
