AI, LLMs, and the Responsible Use of Public Research Data

A Journal (Data) Editor’s Perspective

Lars Vilhuber

2026-11-10

larsvilhuber.github.io/nasem-ai-public-data/

Download slides as PDF

• Dark PDF (white on black)
• Light PDF (black on white, for printing)

Who am I?

Lars Vilhuber

Economist at Cornell University, and the American Economic Association’s Data Editor.

Journals

Data Editor of the AEA

~3000 Manuscripts and 4500 Reports, approx. 4800 authors reached.

DCAP

The question

How can journals ensure scientific integrity, transparency, and reproducibility while also protecting confidentiality in the context of SI/AI-assisted research?

What can journals reasonably require of researchers to promote responsible SI/AI-assisted research?

[record] Session 4, Journal Editor Perspective. Overall theme: AI, LLMs, and the responsible use of public research data.

Three parts

1. What is the same

2. What is different

3. Possible solutions

[record]

Before we start

What are journals interested in?

• Primary: Transparency for all
  • Data access details
  • All code and processing decisions
• Secondary: Reproducibility by many
  • Not all can run code, but many should be able to
  • Does not need to happen before publication

It is understood that not “everybody” can access ALL data.

Access is a scale

Source: https://ukdataservice.ac.uk/help/secure-lab/what-is-the-five-safes-framework/

Access is a scale

An example of access at scale

I will come back to IPUMS later

What is the same

Straw-man arguments

“AI changes everything about reproducibility.”

[record] Set up the straw men. Thesis of the paper (Vilhuber 2026b): the challenges are real, but not new.

“LLM output is non-deterministic, so it cannot be reproduced”

Simulations, sampling, imputation, probabilistic record linkage

Preserve the draws; report the variability

[record] Claim 4 of the paper: generated data are a random draw. Multiple-imputation inference.

“LLM output is non-deterministic, so it cannot be reproduced”

Simulations, sampling, imputation, probabilistic record linkage

Incorporate variability into your analysis (MI, robustness checks, etc.)

“The model is a black box”

Commercial software, proprietary compilers

[record] proprietary compilers (Intel Fortran, PGI, etc.), non-linear optimizers (KNITRO).

“The model is a black box”

Commercial software, proprietary compilers

Document name, version, provider, date

[record] Claim 1: LLMs are code.

“The model is a black box”

Commercial software, proprietary compilers

Show that the proprietary components are not critical for the outcome.

“Black box does not matter”

Ex: Optimum found by quick proprietary solver is also optimum using open source solver

Ex: Open-weight LLM obtains qualitative similar results as proprietary LLM

“The model will change or disappear”

External APIs fail/change, data, websites disappear, deprecated software

“The model will change or disappear”

External APIs fail, data, websites disappear, deprecated software

“The model will change or disappear”

External APIs fail, data, websites disappear, deprecated software

“The model will change or disappear”

External APIs, websites, deprecated software

Preserve the outputs you obtained; document when you accessed the model

[record] Online systems, geocoding, commercial APIs.

“The model will change or disappear”

External APIs, websites, deprecated software

Preserve the software and its environment

“The model will change or disappear”

External APIs, websites, deprecated software

Preserve the open weight LLM

“Training data are confidential or proprietary”

62% of papers already use restricted-access data (Vilhuber 2026a)

[record] Claim 2: LLMs are data. Confidentiality is the norm, not the exception.

“Training data are confidential or proprietary”

62% of papers already use restricted-access data (Vilhuber 2026a)

Data Editors may verify anyway (45% of those)

“Models (and their data) are too big to preserve”

Large data, enclave data, sensitive data

… still looking for easy solutions…

“Models (and their data) are too big to preserve”

Large data, enclave data, sensitive data

(Pietri et al. 2019) 480GB

“Models (and their data) are too big to preserve”

Large data, enclave data, sensitive data

(Survey and Brooks 2022) 83GB, 3500 files

“Models (and their data) are too big to preserve”

Large data, enclave data, sensitive data

(Dell 2023) 340GB

Preserve the generated data

“Models (and their data) are too big to preserve”

Large data, enclave data, sensitive data

Hope that the enclave has a preservation policy.

“Models (and their data) are too big to preserve”

Large data, enclave data, sensitive data

“If you collected the data, preserve it for 5 years.” is not really a solution. There is no easy solution for self-collected sensitive data.

[record] Claim 3: LLMs generate data. Clemens / Census Linking Project / American Stories examples.

“Prompts are something new”

Prompts are code (or metadata)

Include them, with the responses

[record]

So…

None of this is AI-specific, but the difficulties may be magnified when using LLMs in research.

[record] Existing requirements (README, provenance, all code, data where possible) already cover most of this.

What is new

Four new challenges

1. Scale
2. Access mechanisms lagging behind
3. Past practices coming to light
4. Consequence:Shrinking pool of publicly accessible data

[record] None of these are about the LLM per se; they are about the data ecosystem reacting to AI.

1. Scale

Bots everywhere

• Access requests are now easily scaled up
• Previously “freely” accessible resources are struggling with the bandwidth and access requests

Large data sets are increasing

Traditional preservation methods struggle with large data sets

… but more researchers are generating more data.

2. Access mechanisms

Open access

Publicly accessible =

• public domain, or
• CC-BY

• In many cases, no formal recording of agreement/compliance
• In many cases, lack of understanding by economists of what those terms mean.

[record] Distinguish accessible-for-download from re-usable and re-distributable.

Controlled but free access: PSID

• Free, but requires registration
• Conditions of use: no redistribution

• Compliance expected
• Often ignored
• Costly compliance checks (takedown requests) [ link ]

[record] PSID example details. Verify current wording of PSID conditions of use before finalizing.

… and AI raises new questions

• Free, but requires registration
• Conditions of use: no redistribution

• Is uploading data to an AI service redistribution?
• Do terms of use permit AI-assisted processing?

PSID

• Free, but requires registration
• Conditions of use: no redistribution
• Conditions of use: no use of AI

• Reduction in access and availability
• Increase in administrative burden
• Less use

[record]

3. Past practices coming to light

Compliance checks

• Enforcement of terms of use

PSID sent requests to remove PSID data that had been included in replication packages (AEA: 24 cases in 2024)

AI makes scanning cheap

• Existing replication packages can be scanned at scale
• Detect data that should not be there

(Brodeur and Valenta 2026), 327 papers

AI makes scanning cheap

• Existing replication packages can be scanned at scale
• Detect data that should not be there
• Identify poor programming practices

(Schwartz et al. 2026) 1,781 papers Caution: median run-time of 10s not plausible

[record] PII example.

4. Consequences

Consequences

• Researchers have new tools to improve their practices
• Quality can be improved

• Researchers increasingly reticent to share data
• Data custodians increasingly hesitant to share data

[record]

Risks

• Researchers increasingly reticent to share data
• Data custodians increasingly hesitant to share data

Emerging solutions

Building blocks

• Federated university authentication systems
• NIH GA4GH Passports, ICPSR Researcher Passports (Levenstein et al. 2018)
• eduroam!

Building blocks

• APIs are … a thing
• tying APIs to access rights and researcher identity

Unified Access in Social Sciences

• Coordinate on a unified researcher identity across institutions and data repositories
• Use existing mechanisms (not “yet another standard”)
• tie it to access rights and possibly flow control

Caution

• Not too many barriers!
• Not exclusive to academic researchers
  • must allow for other access (hard!)

CANSIM

[record] .

Access mechanisms lagging

Access is still manual

• Web forms, click-through, manual download
• No authenticated APIs for most registered-access data
• Most economists don’t use APIs even when they exist

Access is still manual

• Researchers copy and redistribute rather than re-acquire
• Increasingly bots scrape

[record] AI-assisted (agentic) workflows want programmatic, scripted access.

Exceptions: IPUMS

Solution.

• IPUMS API: authenticated (API key)
• Extract definitions are code ⇒ reproducible acquisition (mostly)
• API key is contingent on accepting terms of use

[record] Terms of use enforced through the authentication, not through the replication package.

Existing NIH practices

• NIH has various cloud-native platforms
• Petabyte-scale data, standardized access protocols

Solution.

• Unified access: GA4GH (Global Alliance for Genomics and Health)
• Distributed computing: commercial and academic clouds

Risk: the pool shrinks

• Data providers do not act as archives
• Centralization creates single-points of failure

[record] Examples?

Other verification methods

Process transparency

TRACE = Transparency Certified

A framework that allows inquiry into the reproducibility workflow at any stage — without requiring re-running the code.

Key insight

Document the process, not just the outputs

• File arrangements (manifests with checksums)
• Processing steps (software, timing, method, isolation)
• Cryptographic signatures by certifying organizations

Why

Transient resources, such as deprecated commercial LLM APIS

When data or services used in computation may no longer be available, TRACE provides a record of what existed at the time of execution.

Generic Workflow

Consider a researcher using confidential data in a Restricted Access Data Center (RADC):

1. Researcher gets environment with confidential data, writes code
2. Code is executed (a) → output produced
3. Data custodian inspects, removes confidential data (b)
4. Researcher receives code + output only (arrangement 3)

Generic workflow with confidential data

TRACE-Compliant

Add a few computationally easy steps:

1. Document each file arrangement (1, 2, 3) with manifests + checksums
2. Describe each processing step (software-driven or manual) with salient info

Example Trusted Research System

TRACE-Compliant

Add a few computationally easy steps:

3. Express in a controlled vocabulary (TROV)
4. Wrap everything into a package with a cryptographic signature → creates a TRO

Example Trusted Research System

TRACE-Compliant

• As of 2026-11-10, there are 40+ TRACE-compliant packages published or in draft form in the AEA Data and Code Repository.
• Confidential data processes at some Restricted Access Data Centers (RADCs) are being adapted to comply with TRACE standards

(Bleemer and Rothstein 2026)

Journals and Transparency

Today

• Provenance and access conditions for all data (incl. data not included)
• All code, including survey code, experimental code
• Generated data, preserved in trusted repositories
• Documentation of how to process (technical, procedural)

Already includes

• prompts (code)
• responses (API, generated data)
• model identifiers and versions (system documentation)
• variability (robustness)

[record] Template README (Vilhuber et al. 2022).

Concerns

• Ability to access resources decreases (expensive APIs)
• Less data available under any access condition

[record]

[record] The confidential-data strategy from the existing materials.

Conclusion

AI is not special

… for reproducibility

… for transparency

AI changes

Scale, size of research

Risk of underprovision of archival resources

AI changes

Programming tools

• Allows for more complex access mechanisms (APIs at scale?)
• Allows for (in principle) better reproducibility

Changing Data Ecosystem

• Less open data
• More enforcement

• Lagging access infrastructure
• Reckoning with past practices

Other?

[record]

Questions

Useful links

• GitHub
• Presentation QR Code:

• Last Run on 2026-11-10

Content is licensed .

References

Bleemer, Zachary, and Jesse Rothstein. 2026. Data and Code for: The Meritocratic Consensus and Stratification in Higher Education. ICPSR - Interuniversity Consortium for Political and Social Research. https://doi.org/10.3886/E259616V1.

Brodeur, Abel, and David Valenta. 2026. On the Prevalence of Personally Identifiable Information (PII) in the Social Sciences. Working {{Paper}} No. 320. I4R Discussion Paper Series. https://www.econstor.eu/handle/10419/343373.

Dell, Melissa. 2023. AmericanStories (Revision 3484aca). Hugging Face. https://doi.org/10.57967/hf/0757.

Levenstein, Margaret C., Allison R. B. Tyler, and Johanna Davidson Bleckman. 2018. The Researcher Passport: Improving Data Access and Confidentiality Protection. May. https://hdl.handle.net/2027.42/143808.

Pietri, Antoine, Diomidis Spinellis, and Stefano Zacchiroli. 2019. “The Software Heritage Graph Dataset.” Version 1.0.0. Zenodo, March. https://doi.org/10.5281/zenodo.2583978.

Schwartz, Matthew, Isaiah Andrews, and Jesse M. Shapiro. 2026. An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research. Working {{Paper}} No. 35782. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w35782.

Survey, U. S. Geological, and Leah Brooks. 2022. “USGS National Map 3DEP 1 Arc-Second Digital Elevation Models (DEMs): Full Coverage for u.s. Interstate Highway System.” Zenodo, January. https://doi.org/10.5281/zenodo.5830968.

Vilhuber, Lars. 2026a. “Report of the AEA Data Editor.” AEA Papers and Proceedings 116 (May): 890–904. https://doi.org/10.1257/pandp.116.890.

Vilhuber, Lars. 2026b. “Reproducibility in an Artificial Intelligence World: Challenges and Solutions for Scientific Research.” https://github.com/larsvilhuber/nasem-ai-public-data/tree/main/text.

Vilhuber, Lars, Marie Connolly, Miklós Koren, Joan Llull, and Peter Morrow. 2022. A Template README for Social Science Replication Packages. Social Science Data Editors. https://doi.org/10.5281/zenodo.7293838.