… when nobody can see your data?

• Because somebody else will access your data, or similar data.
• Because it will actually save you time
• Because if you want to publish, you may need to show reproducibility

• In a typical year, we (AEA) will obtain access (or request access) to 20% of data that is confidential
• Some journals will require that it be provided (privately) and won’t publish otherwise

TIER protocol

TIER Protocol

Structure your project

• Data inputs
• Data outputs
• Code
• Paper/text/etc.

/inputs
/outputs
/code
/paper

/dados/
    /brutos
    /limpos
    /finais
/codigo
/artigo

It doesn’t really matter, as long as it is logical.

TIER Protocol again

TIER Protocol again

TIER Protocol Confidential

TIER Protocol Confidential

/projeto/
     /dados/
        /brutos
        /limpos
        /finais
     /codigo
     /artigo
/secredos            (read-only)
     /impostos       (read-only)
     /salarios       (read-only)

/projeto/
     /dados/
        /brutos
        /limpos
        /finais
     /codigo
     /artigo
/secredos            (read-only)
     /impostos/
              /v1/   (read-only)
              /v2/   (read-only)
     /salarios       (read-only)

How do we handle directories outside of the project space?

DO NOT DO THIS

• “Change the parameter to 0.2, then run the code again”

DO DO THIS

• Use code:
  • Use functions, ado files, programs, macros, subroutines
  • Use loops, parameters, parameter files to call those subroutines
  • Use placeholders (globals, macros, libnames, etc.) for common locations ($CONFDATA, $TABLES, $CODE)
• Compute all numbers in package
  • No manual calculation of numbers

DO NOT DO THIS

• “The figure will display in a window, save it to the paper directory”
• “The logfile will show the regression coeffs, copy them into the Excel sheet”

DO DO THIS

• Dynamic documents\(^*\)
• Write out/save tables and figures using packages
• Use functionality to write directly into Excel sheets

• No manual install of packages
  • Use a script to create all directories, install all necessary packages/requirements/etc.
  • Use tools like renv (R), requirements.txt (Python), project.toml (Julia)
  • Write all Stata packages into a project-specific directory and do not re-install

// Header of main.do

global rootdir : pwd
global adodir  "${rootdir}/ado"
cap mkdir "$adodir"
adopath ++ "$adodir"
adopath - PLUS     // Just to be sure
adopath - PERSONAL // Just to be sure
cap ssc install estout // this will now go into the new adodir we created!

• Controller script (main.do, run.sh, makefile) preferred
  • Least amount of manual effort
• Clear instructions

Repeatedly running through the entire sequence ensures later robustness

• Presumes a single control file (main.do, run.sh)
• Presumes ability to version results!
• Leverage (data provider-specific) tools, such as job queues
• Use free resources
  • Job queues with low priority
  • Human downtimes (leaving for a holiday, going to a conference)

Introduce break points

• Do not reprocess when no changes to data or code happen
• Always reprocess if data or code change
• Decision might be
  • Manual (flags)
  • Automated (make, checksums, etc.)

Introduce break points: manual

// Header of main.do
global step1 0
global step2 1
global step3 1

if $step1 == 1  do "$code/01_extract.do"
if $step2 == 1  do "$code/02_analysis.do"
if $step3 == 1  do "$code/03_figures.do"

Introduce break points: automated

// Header of main.do
global step1 0
global step2 1

// verify if file has changed
global checksum1 386698503
qui checksum "$resultfile1"
if `r(checksum)' == $checksum1 global step2 0 

if $step1 == 1  do "$code/01_extract.do"
if $step2 == 1  do "$code/02_analysis.do"
if $step3 == 1  do "$code/03_figures.do"

(or use make, snakemake, cmake, …)

(Frequency in AEA journals: 1-2 per year…)

Use configuration files

As file structure becomes more complex, configure short-cuts (globals, variables, etc.)

// config.do
global outputdata "/projeto/dados/limpos"   // this is where you would write the data you create in this project
global results    "/projeto/artigo"         // All tables for inclusion in your paper go here
global programs   "/projeto/codigo"         // All programs (which you might "include") are to be found here

• In the United States, some variables on IRS databases are considered super-top-secret. So you can’t name that-variable-that-you-filled-out-on-your-annual-tax-Form-1040 in your analysis code of same data. (They are often referred to in jargon as “Title 26 variables”).
• Your code contains the random seed you used to anonymize the sensitive identifiers. This might allow to reverse-engineer the anonymization, and is not a good idea to publish.
• You used a look-up table hard-coded in your Stata code to anonymize the sensitive identifiers (replace anoncounty=1 if county="Tompkins, NY"). A really bad idea, but yes, you probably want to hide that.
• Your IT specialist or disclosure officer thinks publishing the exact path to your copy of the confidential 2010 Census data, e.g., “/data/census/2010”, is a security risk and refuses to let that code through.
• You have adhered to disclosure rules, but for some reason, the precise minimum cell size is a confidential parameter.

So whether reasonable or not, this is an issue. How do you do that, without messing up the code, or spending hours redacting your code?

• This will serve as an example. None of this is specific to Stata, and the solutions for R, Python, Julia, Matlab, etc. are all quite similar.
• Assume that variables q2f and q3e are considered confidential by some rule, and that the minimum cell size 10 is also confidential.

set seed 12345
use q2f q3e county using "/data/economic/cmf2012/extract.dta", clear
gen logprofit = log(q2f)
by county: collapse (count)  n=q3e (mean) logprofit
drop if n<10
graph twoway n logprofit

A bad example, because literally making more work for you and for future replicators, is to manually redact the confidential information with text that is not legitimate code:

set seed NNNNN
use <removed vars> county using "<removed path>", clear
gen logprofit = log(XXXX)
by county: collapse (count)  n=XXXX (mean) logprofit
drop if n<XXXX
graph twoway n logprofit

The redacted program above will no longer run, and will be very tedious to un-redact if a subsequent replicator obtains legitimate access to the confidential data.

Note that you have to re-run the entire code to obtain a modified graph, e.g., if you want to add some reference line, or change colors. But if the data presented in the graph is non-sensitive (i.e., disclosable), then the data underlying it is as well. Thus, and this is a more general approach, we can provide code that automatically detects if the confidential data is there, and only then will it run the data preparation part, but it will always run for the graphing (“analysis”) part of the code.

We also introduce the use of a separate file for all the confidential parameters, which may be more convenient, since now, no redaction is needed - the confidential file is simply dropped (but should be documented).

Thus, the replication package would have:

main.do
README.md
config.do
include/confparms_template.do
releasable/figure1.dta
releasable/figure1.pdf

• Using templates for reproducibility: fabulous template README
• Documenting what you did: fill out the README
• When to document: constantly

// Header of main.do

include "config.do"

global step1 0
global step2 1

// verify if file has changed
qui checksum "$resultfile1"
if `r(checksum)' == $checksum1 global step2 0 

if $step1 == 1  do "$code/01_extract.do"
if $step2 == 1  do "$code/02_analysis.do"
if $step3 == 1  do "$code/03_figures.do"

// Header of main.do

include "config.do"
    //               global extractpct 1

global step1 0
global step2 1

// verify if file has changed
qui checksum "$resultfile1"
if `r(checksum)' == $checksum1 global step2 0 

if $step1 == 1  do "$code/01_extract.do" $extractpct
if $step2 == 1  do "$code/02_analysis.do"
if $step3 == 1  do "$code/03_figures.do"

// Header of main.do

include "config.do"
    //               global extractpct 1

global step1 0
global step2 1

// verify if file has changed
qui checksum "$resultfile1"
if `r(checksum)' == $checksum1 global step2 0 

if $step1 == 1  do "$code/01_extract.do" $extractpct  // extract from BIGDATA
if $step2 == 1  do "$code/02_analysis.do"
if $step3 == 1  do "$code/03_figures.do"

As we test our code, we use only a small fraction:

    //               global extractpct 1

We save the extract in $resultfile1, on which we can compute a checksum

qui checksum "$resultfile1"

and if that hasn’t changed, we don’t do the merge stuff again:

if `r(checksum)' == $checksum1 global step2 0 
if $step2 == 1  do "$code/02_analysis.do"

• While we are testing, only small chunk
• Once we are done testing, we change one parameter (maybe two), and reprocess the whole thing again! Without much human work!

True.

(assume a job scheduler of some sort, with a counter)

// Header of main.do

include "config.do"
    //               global extractpct 1
global step1 0
global step2 1

// verify if file has changed
qui checksum "$resultfile1"
if `r(checksum)' == $checksum1 global step2 0 

if $step1 == 1  do "$code/01_extract.do" $extractpct  // extract from BIGDATA
if $step2 == 1  do "$code/02_analysis.do"
if $step3 == 1  do "$code/03_figures.do"

(let’s change the location of the file check)

// Header of main.do

include "config.do"
    //               global extractpct 1
global step1 0
global step2 1

if $step1 == 1  do "$code/01_extract.do" $extractpct  // extract from BIGDATA

// verify if file has changed
qui checksum "$resultfile1"
if `r(checksum)' == $checksum1 global step2 0 

if $step2 == 1  do "$code/02_analysis.do"

(Let’s allow for an additional parameter to be passed)

// Header of main.do
global parm `1'      // <--- this grabs any parameter we pass on launch

include "config.do"
    //               global extractpct 1
global step1 0
global step2 1

if $extractpct != $parm {        // <-- If we pass a non-std value
      global step1 1             // <-- we force step1 to be executed
      global extractpct $parm    // <-- with the new parameter
}
if $step1 == 1  do "$code/01_extract.do" $extractpct  // extract from BIGDATA

// verify if file has changed
qui checksum "$resultfile1"
if `r(checksum)' == $checksum1 global step2 0 
                                 // <-- and this will then fail, kicking in step2
if $step2 == 1  do "$code/02_analysis.do"

This can be launched in a loop:

for ITER in $(seq 1 100)
do
  stata do main.do $ITER &
done

which will launch 100 jobs simultaneously!

BETTER: Use a tool like qsub, slurm or similar that can also use a loop, and ship off jobs to multiple compute nodes (check your local HPC cluster)

We will still need program accordingly:

• $code/02_analysis.do needs to save the output accordingly (not overwrite)
• logfiles need to be collected, should something go wrong
• A downstream program will need to collect the results again.
• Talk to your sysadmin first before you overwhelm a system!

What happens when I (or anybody else) cannot access the data you used?

• The data is available from private provider to you only
• You have to be physically present in a particular location
• The data have been deleted (e.g. legal requirements)
• The data have been modified since you used them (e.g, GDPR, Twitter)
• The data are too big to move to a different location (big data in cloud)

• In “classical” replication package:

I don't trust you, therefore give me data, code, I will run myself.

• Provides some evidence that complete package, same results
• Inefficient
• Here?

I (Lars Vilhuber) know that I can trust cascad.tech when they run replication packages for me because

• We have discussed in detail about the process (inspection)
• I have interacted with them multiple times (verification)

cascad website

But nobody can solve the problem in all scenarios, especially when data are one-time.

We have to verify live, once, and convey that.

“TRAnsparency CErtified (TRACE): Trusting Computational Research Without Repeating It”

nsf search result

• No researcher interaction during running
• A standardized, transparent way of describing HOW things are run
• Ability to verify integrity of replication package as created the first time
• A few other things (that still need to be defined)

• recorded runs from services like Wholetale and Codeocean – need a bit of description, might need “isolated” (airgapped) runs, cannot handle confidential data
• Institutions like BPLIM and IAB FDZ where reproducibility is baked in – code is run by systems or non-author staff – need more transparency into the process, need a way to lock in the output packages, can’t package up the input data

In both cases, individual inspection and conversations can lead to trust, similar to cascad, but not scalable, and “leaky”.

• a way of describing a system - any system, from
  • launching code,
  • to accessing data,
  • to collecting output,
  • including any non-researcher inputs (editing/ confidentiality)

Key

• the researcher cannot manipulate the system.
• the output, if not all the inputs, needs to be bundled up in a credible way.
• comparable across systems: Wholetale, Codeocean, cascad, university or central bank HPC cluster, FSRDC, IAB, etc.
• must be able to handle confidential inputs and intermediate outputs, flow data, BIG DATA

Then, once you’ve described a system, you need to certify a “run” of the system.

That’s the next piece: Transparency-Certified Research Objects (TROs)

• Sketch of key description
• Sample implementation (simplest software and case)

Goals:

• Specifications that anybody can implement (it is NOT a software implementation)
• But also: sample software implementations (for SLURM job scheduler, for instance, but also Wholetale recorded runs)