Reproducibility Interships: Goals and Outcomes
Lars Vilhuber
2024-10-07
Richard Ball asked the (rhetorical) question “Is it feasible to include reproducible research methods in undergraduate training?” (Ball 2023), answering that question with “Yes we can!”.
I will go beyond that, and argue “Yes we should!”
Why?
There is a lot of opportunity
- As of September 2024, the LDI Replication Lab has verified around 2500 distinct AEA journal manuscripts as to their computational reproducibility. (Vilhuber 2024)
This is being done…
- Since 2014 (starting before the official AEA work), more than 200 undergraduates have been trained, and have helped in the assessment.
… But not in the classroom (mostly)
- The training is not part of a regular (Cornell) curriculum.
What is a replication package?
Student involvement
What do the students do?
- Analyze data provenance as described by authors
- Verify ability to computationally reproduce results
- Attempt reproduction as per instructions (README)
- Inevitably debug
- Write a shareable report
How are the students trained?
- Intense 1-day training on principles and technology
- Loose follow-up on three examples
- Then real cases.
What are students trained on?
- Technological reasons (use of specific workflow management system, other tools)
- Versioning: All reproducibility checks are versioned (git crucial)
- Running code (R, Stata, Python, Julia, Matlab, SAS, …)
Communication
- Debugging: There is always some … and reproducibly debugging is important (internal communication)
- Writing reports: external communication to authors, conveying what was done, and what may have gone wrong (constructive and objective criticism)
Organization
- Organizing their work within the Lab
- Structuring reports and other documents
Is this useful?
Students think so
2020 sociology graduate working for a nonprofit research organization
“[I received] overwhelmingly positive feedback on my documentation method in code reviews, which is all thanks to my time with LDI”
2021 Economics graduate, currently pursuing a Ph.D. in Political Economics
“… I feel like [LDI Replicator position] was the single most important thing … to prepare myself to succeed in [predoctoral fellowship]….”
2024 Economics intern
For every issue I did solve, there was a gratifying moment, knowing I’ve explored something new and that the authors would read and heed my documented solution. It’s always a huge confidence boost whenever I take the initiative to research and dabble with problems I might not be able to solve.
Summer 2024 internship at AEA
Participants
- Initiated by Cornell University (Vilhuber)
- 5 partner institutions (Wellesley, Haverford, Bryn Mawr, Notre Dame, U Colorado at Boulder, UNC Chapel Hill\(^*\)), each with a local coordinator
- 9 students selected in total
Timeline
- Training in April (to skirt finals)
- Start after finals (May 20 - June 1, depending)
- 12 weeks core participation
Structure
- 12-16 weeks of participation
- Funded by journal for real work (real articles, real reports)
- Partially supported by various host institutions
Student contributions
- Part of regular team (others were continuing Cornell students)
- 12-16 “cases” (~ 1/week)
- A small number of revisions (same article, improved replication package)
Student learning
- Successful debugging (with assistance when needed)
- Structured internal reporting (how to convey what you did that did not work)
- Writing reports
Student learning (technical)
- Use of remote Windows and in some cases Linux servers
- Basic use of multiple software not previously used
- Repeated use of modern software development tools: Git, Markdown, automation
Student experiences and reproducibility services
Student experiences and reproducibility services
- Journals are organizationally distinct from universities (mostly)
Student experiences and reproducibility services
- Most curricula currently do not include the methods and tools necessary for reproducible work
Student experiences and reproducibility services
- Most research institutions to not provide reproducibility services to their researchers
Timing of publication
- In the context of the AEA journals, students provided feedback prior to (first) publication
- World Bank, some research institutes are providing service to their staff / members with public artifacts (early institute-level publication)1
- i4R is providing more in-depth feedback post-publication
Yes we should
Yes we should
- Include training on reproducible practices in undergraduate curricula (junior / penultimate year) 2
- Training should include practice on pre-publication (own faculty) and post-publication (journal publications)
- May involve structured reporting (Social Science Reproduction Platform)
Thank you.
References
Ball, Richard. 2023. “‘Yes We Can!’: A Practical Approach to Teaching Reproducibility to Undergraduates.” Harvard Data Science Review 5 (3). https://doi.org/10.1162/99608f92.9e002f7b.
Butler, Courtney R. 2023. “Publishing Replication Packages: Insights From the Federal Reserve Bank of Kansas City.” Harvard Data Science Review 5 (3). https://doi.org/10.1162/99608f92.aba61304.
Mendez-Carbajo, Diego, and Alejandro Dellachiesa. 2023. “Data Citations and Reproducibility in the Undergraduate Curriculum.” Harvard Data Science Review 5 (3). https://doi.org/10.1162/99608f92.c2835391.
Peer, Limor. 2024. “Why and How We Share Reproducible Research at Yale University’s Institution for Social and Policy Studies.” Harvard Data Science Review 6 (1). https://doi.org/10.1162/99608f92.dca148ba.
Vilhuber, Lars. 2024. “Report of the AEA Data Editor.” AEA Papers and Proceedings 114 (May): 878–90. https://doi.org/10.1257/pandp.114.878.
Vilhuber, Lars, Hyuk Harry Son, Meredith Welch, David N. Wasser, and Michael Darisse. 2022. “Teaching for Large-Scale Reproducibility Verification.” Journal of Statistics and Data Science Education 30 (3): 274–81. https://doi.org/10.1080/26939169.2022.2074582.