Tutorial reproducibility
Please fill out this survey on background and skills, to provide us with information on who you are. It will help us improve the presentation, and make it more relevant for you.
https://cornell.yul1.qualtrics.com/jfe/form/SV_bBqbJ9cSSJdOBw2
One of the following (or a linear combination):
Results from Survey 1
Last updated: 24 November 2025, 14:34
The results cover multiple workshops, and include all responses between 01 January 2025 and 01 December 2025.
Education
| Degree | Frequency | Frequency (Complete) | Percent |
|---|---|---|---|
| graduate student (Ph.D.) | 26 | 26 | 20.16 |
| graduate student (Masters) | 10 | 8 | 6.20 |
| faculty member (tenured) | 15 | 12 | 9.30 |
| faculty member (untenured) | 9 | 8 | 6.20 |
| Other | 60 | 51 | 39.53 |
| NA | 9 | 0 | 0.00 |
| Note: | |||
| Percent is calculated as the proportion of completed responses. |
Operating System by Educational Category
| OS | Academic | Other | Overall |
|---|---|---|---|
| Linux | 9.4% | 13.1% | 11.2% |
| Mac | 45.3% | 9.8% | 28.0% |
| Windows | 45.3% | 77.0% | 60.8% |
| Note: | |||
| Completed responses only. Multiple mentions possible. Cells show percent of OS mentions by group. |
Intermediate or Advanced Programming Language Usage
| Programming Language | Frequency | Percent |
|---|---|---|
| C | 18 | 17.14 |
| Matlab | 15 | 14.29 |
| Python | 33 | 31.43 |
| R | 53 | 50.48 |
| SAS | 9 | 8.57 |
| Stata | 63 | 60.00 |
| Note: | ||
| Completed responses only. Multiple mentions possible. |
Command Line Usage by Educational Category
| Feature | Academic | Other | Overall |
|---|---|---|---|
| Command line used often | 48.5% | 43.1% | 61.0% |
| Command line used once | 16.2% | 16.7% | 21.9% |
| System with >6 CPUs: often | 25.0% | 31.9% | 38.1% |
| System with >6 CPUs: once | 10.3% | 8.3% | 12.4% |
| Note: | |||
| Completed responses only. Cells show percent of feature mentions by group. |
One of the following (or a linear combination):
- Reproducibility when some data are confidential
- Advanced self-checking of reproducibility
- Preserving raw survey data early in research lifecycle (ethically!)
Survey 2
To identify what we should be speaking about on Day 2, please fill out this other survey:
https://cornell.yul1.qualtrics.com/jfe/form/SV_cNkhKL69K2Ob7o2
Results from Survey 2
Last updated: 24 November 2025, 14:34
The results cover multiple workshops, and include all responses between 01 January 2025 and 01 December 2025.
| Preferred_topic | Frequency | Percent |
|---|---|---|
| Advanced self-checking of reproducibility | 10 | 34.48 |
| Preserving raw survey data | 6 | 20.69 |
| Reproducibility when some data are confidential | 11 | 37.93 |
| NA | 2 | 6.90 |
The last day serves to review the various materials, handle any questions not addressed (in detail) on the previous days, and discuss experiences and difficulties applying these principles in your work.
Topics may include:
- Advanced self-checking of reproducibility (the more technical parts)
- Preserving raw survey data early in research lifecycle (ethically!)
- Reproducibility for LLM and AI
Guidance
Some additional guidance can be found on the website of the Social Science Data Editors (URLs subject to change):
Examples of replication packages
With confidential data
- https://doi.org/10.3886/E154241V2 not only code, but faces the problem that IRS data cannot have variables revealed. Their workaround is not the same one as in this tutorial.
- https://doi.org/10.3886/E162581V1
Using containers:
- Kline et al (2024) “A Discrimination Report Card: primary replication package, with container specification, image on Docker Hub, and preserved image on Zenodo.
- Herbert et al (2024) “Reproduce to validate”: primary replication package, container specification and preserved image on Borealis.ca.
Extra info
- This document’s source: https://github.com/larsvilhuber/tutorial-reproducibility-2025
- Licensed under
