Reproducibility in research – Ensuring the transparency and credibility of your work

Lars Vilhuber

Marie Connolly

2026-05-26

Follow along

larsvilhuber.github.io/workshop-uppsala-2026/presentation/presentation.html (PDF)

Who am I?

Lars Vilhuber

Executive Director of the Labor Dynamics Institute and Senior Research Associate in the Economics Department at Cornell University, and the American Economic Association’s Data Editor.

Data Editor of the AEA

2389 Manuscripts and 4440 Reports, approx. 4400 authors reached.

Walkthrough of the agenda

Today

9:00 Welcome
9:05 Walkthrough
9:15 Goals
9:30 🔒 Hands-on Exercise: A very imperfect example
9:45 Discussion of the “Very imperfect example”
10:00 Setting yourself up for reproducibility from Day 1
10:30 Break
10:45 How to correctly document a replication package (and why!)
11:15 Reproducibility when some data are confidential
11:45 🔒 Explanation of “homework”

Day 2

9:00 Review of “homework”, assignment of reviewers, active review
9:30 Discussion of results from review
10:00 Advanced reproducibility setup
10:45 Break
11:00 Choices:
- Provenance and preservation
- APIs and principled retrieval (with SIVACOR)
11:45 Wrap up

Best practices?

First: why?

Why reproducibility?

Credibility
Transparency (openness)
Efficiency of scholarly discourse?

Why reproducibility?

Early publications (20th century) contained tables of data, and the math was simple (maybe)
Data became electronic, were no longer included or cited
Math was transcribed to code, and was no longer included

Increasing broad consensus in academia

FAIR principles
Data Citation Principles
Computational Reproducibility

Findable
Accessible
Interoperable
Reusable

Data Citation Principles

FAIR principles
Data Citation Principles
Computational Reproducibility

To make it findable, citations,
Give attribution and credit for data.

Computational reproducibility

FAIR principles
Data Citation Principles
Computational Reproducibility

Primary topic today

Reproducibility means obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis.²

What is…

What is a replication package?

A Replication Package is

Code
Data
Materials (for surveys, experiments, …)
Instructions on how to obtain data not included
Instructions on how to combine it all
Known issues documented

Complies with…

Is stored in…

AEA Data and Code Repository
Other trusted repositories

Best practices?

Summing up

Why
- Credibility
- Transparency (openness)
- Efficiency of scholarly discourse ([example])
How
- FAIR principles
- Data Citation Principles
- Computational Reproducibility

As Replication Packages
- Code
- Data
- Materials (for surveys, experiments, …)
- Instructions on how to obtain data not included
- Instructions on how to combine it all
- Known issues documented

Who?

🐇 Authors at conditional acceptance
🐢 Authors at submission
🐁 Authors at beginning of project

👴🏻👵🏽 Experienced researchers
👶🏽👶🏻 Junior researchers
👨‍🎓👩‍🎓 Ph.D. students
🧒👦 Undergraduates

Who?

You.

You

👶🏻 Now:

more efficient development
more efficient collaboration
more assurance that “everything just works”

👵🏽 Soon

more efficient development across projects
more efficient response to editors and referrees, for any paper

You.

… while you are in a new institution, on a new computer, with three courses to prep, a new baby, and searching for a house big enough
and (luxury!) a RA you can delegate to…

You?

How?

How to create reproducible research?

Habits

Reproducibility from Day 1
Adopt reproducible habits
Take notes when you do things, not after
Use version control

Strategy

Computational empathy: think of the next person to run this - It could be you in 5 years!

Hands-on: A very imperfect example

Presentation of the example

Day 1 reproducibility

Day 1: How to not to organize your work

Contributed by Michael Stepner, with anonymity of the counter party preserved…

Have you had this conversation?

Day 1: Setting yourself up for reproducibility

An approach to be reproducible from Day 1

Break

Day 1: Document it

How to correctly document a replication package (and why!)

Day 1: But my data are confidential

Reproducibility when some data are confidential

End of Day 1

Day 2

Appendix

Where to?

Choices

Issue No. 1: Data Access and preservation 💥 [a]
Issue No. 2: Confidential data - same ⬆️! [a] [b]
Lifecycle checking: Self-checking reproducibility and presentation
New challenges: AI and Big Data but same ⬆️!
New methods: Transparency outsourced or certified
Implementation in academia: Students!

Resources

README

Lars Vilhuber, Connolly, M., Koren, M., Llull, J., & Morrow, P. (2022). A template README for social science replication packages (v1.1). Social Science Data Editors. https://doi.org/10.5281/zenodo.7293838

You can download the Word, LaTeX, or Markdown version of the README with lots of examples.

Other guidance

Presentation on “Self Checking Reproducibility” and its associated website
Guidance when (some) data are confidential: https://labordynamicsinstitute.github.io/reproducibility-confidential/
Guidance for citations: https://social-science-data-editors.github.io/guidance/addtl-data-citation-guidance.html

Extra info

This document’s source: https://github.com/larsvilhuber/workshop-uppsala-2026
Licensed under

Sources

Images: NYT, Bluesky 1, 2, Ike Hayman/Wikimedia

Details on Transparency, etc.

Transparency

Provenance of the data
Processing of the data, from raw data to results (code)

It is the policy of the American Economic Association to publish papers only if the data used in the analysis are clearly and precisely documented and access to the data and code is clearly and precisely documented and is non-exclusive to the authors.

Completeness

All data needs to be identified and and access described
All code needs to be described and provided
All materials must be provided (survey forms, etc.)

Authors … must provide, prior to acceptance, the data, programs, and other details of the computations sufficient to permit replication

Preservation

All data needs to be preserved for future replicators
- Ideally, within the replication package, subject to ToU, for convenience
- Otherwise, in a trusted repository

Preservation

Code must be in a trusted repository
- Usually, within the replication package
- Websites, Github, are not acceptable

Historically

AER 1911 thanks to Stefano Dellavigna

Modern preservation

Exceptions to the Policy

None

…

… there is a grey zone:

When data do not belong to researcher, no control over preservation, access!
Sometimes, ToU prevent researcher from revealing metadata (name of company, location)

Transparency again

However:
- No exception for need to describe access (own and other)
- No exception for need to fully describe processing (possibly with redacted code)

Reproducibility in Economics and beyond

Data Editors

American Economic Association (8)
Econometric Society (3)
Canadian Journal of Economics (1)
Royal Economic Society (2)
Western Economic Association International (1)
European Economic Association (1)
Review of Economic Studies (1)

Common policies

https://social-science-data-editors.github.io/

Elsewhere: Political Science

Elsewhere: Sociology

But!

Elsewhere: Sociology

Weeden (2023) ³

Benefits

Building on the work of others

Roth, Jonathan. 2022. “Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends.” American Economic Review: Insights 4 (3): 305–22. DOI: 10.1257/aeri.20210236

Notes: “I exclude 43 papers for which data to replicate the main event-study plot were unavailable.”

Building on the work of others: dCdH 2020

de Chaisemartin, Clément, and Xavier D’Haultfœuille. 2020. “Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects.” American Economic Review 110 (9): 2964–96. DOI: 10.1257/aer.20181169

The results from various other papers are recomputed to empirically demonstrate the relevance of the proposed methods.

Transparency elsewhere

Transparency outsourced

A third party conducts the reproducibility, not you, not me.
Need to common understanding, protocols, etc.
AEA’s protocol
We do this about a dozen times per year

Transparency outsourced

Why should I believe the third party?

Trust
Transparency
Common methods

Transparency certified

Transparency certified

Providing information about the computing platforms themselves, including specific details about how computational transparency is supported.
Packaging and signing resulting artifacts along with records of their execution using a standard format.

Try it out

SIVACOR

Footnotes

Data Citation Synthesis Group: Joint Declaration of Data Citation Principles. Martone M. (ed.) San Diego CA: FORCE11; 2014 https://www.force11.org/group/joint-declaration-data-citation-principles-final
National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303.
Weeden, K. A. (2023). Crisis? What Crisis? Sociology’s Slow Progress Toward Scientific Transparency . Harvard Data Science Review, 5(4). https://doi.org/10.1162/99608f92.151c41e3