Open Science in Economics

Lars Vilhuber

2026-07-09

Follow along

larsvilhuber.github.io/zbw-open-science-2026/

Who am I?

Lars Vilhuber

Executive Director of the Labor Dynamics Institute and Research Professor in the Economics Department at Cornell University, and the American Economic Association’s Data Editor.

Lars Vilhuber

Data Editor of the AEA

2791 Manuscripts and 4440 Reports, approx. 4900 authors reached.

Journals

What is Open Science?

What is Open Science?

Usually described along several pillars:

  • Open access to publications
  • Open data
  • Open code / software
  • Open methods and peer review

Open Science and Credibility

Open Science facets are seen as good, and a (possibly) necessary part of credible research

Open Science and Policy

Should policy be based on credible research, and thus on open science?

Why Data Access Is the Hard Part

  • Manuscripts: increasingly open (preprints, open access mandates)
  • Code: cheap to share, easy to version, low legal risk
  • Data: the sticking point

Recent Evidence

A 2026 issue of Nature on social science reproducibility:

  • Headline: “Half of social-science studies fail replication test”
    • But: only 30% of articles yielded data

Fig 2, Tyne et al. (2026)

Let’s make that a bit bigger

Fig 2, Tyne et al. (2026), data from 2009 to 2018

Why is data not always open?

  • Data withheld for ethical, legal, or contractual reasons
  • May be confidential, proprietary, or administrative

Ask yourself!

Would you want

  • your complete medical history
  • your precise address

to be made public?

Probably not

So: Open Science and Policy

Should policy be based on credible research, and thus on open science,…

when

open science == open data?

Open Access and Politics

But this is exactly what certain critics of environmental policies have suggested in the US.

Open Access and Politics

No health data… no health effects of pollution… no “costly” environmental regulations

Data Access as the Bottleneck

  • Access — not mere existence — determines whether findings can be verified

Without access to the underlying data, code and methods alone cannot establish reproducibility — verification becomes much harder.

How do you do open science when data are not “freely available”?

Shades of access

The Problem

Recent Evidence

A 2026 issue of Nature on social science reproducibility:

  • Headline: “Half of social-science studies fail replication test”
    • But: only 30% of articles yielded data; only 24% were attempted

Fig 2, Tyne et al. (2026)

Recent Evidence

A 2026 issue of Nature on social science reproducibility:

  • Headline: “Half of social-science studies fail replication test”
    • But: only 30% of articles yielded data; only 24% were attempted
    • 74% of those attempted were exactly or approximately reproducible

Fig 2, Tyne et al. (2026)

Enormous effort

A 2026 issue of Nature on social science reproducibility:

  • Brodeur et al. (2026), crowdsourced:
    • 85% of 110 articles (2022–2023) were reproducible
    • Required 80 replication games and 3,500+ researchers

Fig 2, Brodeur et al. (2026)

But: Restricted Data Remains Unassessed

None of these studies could access restricted data — entire swaths of social science literature remain outside scope.

The Scale of the Challenge

From the AEA Data Editor’s experience (2025, 384 papers assessed):

  • 38% used data with no access restrictions — in scope for replication studies
  • 62% used data subject to access restrictions

Access categories and data sharing

The Scale of the Challenge

Of those restricted papers:

  • The AEA team obtained private access to 45% of the 62%
  • Conducted reproducibility checks despite data not being in public packages

Access categories and data sharing

An information and trust problem

An information and trust problem

  • Verification at large cannot ascertain whether replication packages are reproducible.
  • Yet many of those same packages are reproducible.
  • That fact might rely on trust in data editors, or others.

The Key Question

What if it were possible to credibly demonstrate that the original execution of the computational artifacts occurred in a transparent fashion, even when data cannot be published, and is consistent with the deposited computational artifacts (code) and outputs (figures and tables)?

The Key Question

What if it were not necessary to re-run the code?

Certification

If reproducibility can be certified at the source, then:

  • Readers can trust results and focus on robustness, not reproduction
  • Researchers can convey credibility even when data cannot be shared
  • The scale of verification effort shrinks dramatically

Certification is not new

Services now provide active verification:

Service Approach
cascad Certification service, access to confidential data
World Bank Internal service, access to confidential data
Codeocean Containerized capsules with manual compliance checking

The Transparency Gap

When a reproducibility service has re-run code and issued a certificate — what actually happened?

The Transparency Gap

Questions that remain unanswered:

  • Was work done with input/influence from authors, or in isolation?
  • Was internet access available during the run?
  • What state was a database in when first queried vs. when the service ran it?
  • Was code modified (inadvertently or intentionally) by verifiers?

Absent standardized protocols or vocabularies, services remain opaque.

Readers, journals, and researchers cannot compare:

  • Codeocean vs. World Bank
  • cascad vs. AEA

The TRACE Framework

What is TRACE?

TRACE = Transparency Certified

A framework that allows inquiry into the reproducibility workflow at any stage — without requiring re-running the code.

Key insight

Document the process, not just the outputs

  • File arrangements (manifests with checksums)
  • Processing steps (software, timing, method, isolation)
  • Cryptographic signatures by certifying organizations

Result

TRACE-compliant packages can be:

  • Compared across services (Codeocean vs. World Bank)
  • Inspected both by humans and automated scripts
  • Trusted via organizational credibility chains

Generic Workflow (Before TRACE)

Consider a researcher using confidential data in a Restricted Access Data Center (RADC):

  1. Researcher gets environment with confidential data, writes code

Generic workflow with confidential data

Generic Workflow (Before TRACE)

Consider a researcher using confidential data in a Restricted Access Data Center (RADC):

  1. Researcher gets environment with confidential data, writes code
  2. Code is executed (a) → output produced

Generic workflow with confidential data

Generic Workflow (Before TRACE)

Consider a researcher using confidential data in a Restricted Access Data Center (RADC):

  1. Researcher gets environment with confidential data, writes code
  2. Code is executed (a) → output produced
  3. Data custodian inspects, removes confidential data (b)
  4. Researcher receives code + output only (arrangement 3)

Generic workflow with confidential data

Generic Workflow (Before TRACE)

Problem:

  • The World Bank certificate states in English that this process was followed.
  • Codeocean points to an FAQ.

Neither is verifiable or machine-readable.

Generic workflow with confidential data

Making a Workflow TRACE-Compliant

Making a Workflow TRACE-Compliant

Add a few computationally easy steps:

  1. Document each file arrangement (1, 2, 3) with manifests + checksums
  2. Describe each processing step (software-driven or manual) with salient info

Example Trusted Research System

Making a Workflow TRACE-Compliant

Add a few computationally easy steps:

  1. Express in a controlled vocabulary (TROV)
  2. Wrap everything into a package with a cryptographic signature → creates a TRO

Example Trusted Research System

Scenarios TRACE Addresses

SCN3 — Restricted access data environments

RADCs have no vested interest in any particular paper — they satisfy the arms-length requirement.

Partners at central banks (and World Bank!) are already implementing TRACE-compliant capabilities.

Thank you

Additional info

Further reading

Schmidt, Klaus M., Levent Neyse, Marianne Saam, Doreen Siegfried, Lars Vilhuber, and Joachim Winter. 2026. “Open Science in Den Wirtschaftswissenschaften: Transparenz, Reproduzierbarkeit Und Zugang.” Perspektiven Der Wirtschaftspolitik. https://doi.org/10.1515/pwp-2026-0019.

Vilhuber, Lars. 2025. “Reproducibility and Open Science in Economics.” Revue Économique 76 (5). https://doi.org/10.3917/reco.765.0697.

Appendix