Testing Identifying Assumptions in Examiner IV Designs: A Vignette Experiment with Polish Prosecutors

How to cite

Sigstad, Henrik; Šoltés, Michal; Uhl, Andrzej. 2026. "Testing Identifying Assumptions in Examiner IV Designs: A Vignette Experiment with Polish Prosecutors." AEA RCT Registry. AEARCTR-0018783. https://doi.org/10.1257/rct.18783-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Henrik Sigstad* , Michal Šoltés , Andrzej Uhl

* Primary investigator

Registry ID
AEARCTR-0018783
DOI
https://doi.org/10.1257/rct.18783-1.1
First registered
Last updated
Status
In development
Trial dates
2026-06-18 – 2026-07-15
Countries
Private
JEL codes
C26, C93, K14, K42
Registry record
http://www.socialscienceregistry.org/trials/18783

Abstract

Many empirical studies in economics, criminology, and public policy use the random assignment of cases to decision-makers—judges, prosecutors, bail magistrates, disability examiners, patent reviewers, and caseworkers—to identify the causal effects of their decisions. These “examiner IV” designs rely on a monotonicity assumption: examiners who are stricter on average must be weakly stricter for every individual case. Yet this assumption is essentially untestable in administrative data, because the same case is rarely decided by multiple examiners.

This study conducts a vignette experiment with up to 250 Polish public prosecutors, each of whom independently recommends a sentence, including both type and severity, for the same fixed set of criminal cases. Observing many prosecutors’ decisions on identical cases allows us to measure how often monotonicity fails when decisions are made independently, without the panel-deliberation dynamics that complicate prior evidence from judicial panels. It also allows us to compute directly the case-level weights that determine whether the standard practice of controlling for non-focal decision propensities yields a well-defined estimand when examiners make multidimensional choices.

Prosecutors are randomized into four groups that vary in the homogeneity of their case mix, ranging from all-shoplifting cases to a diverse set of crime types. This design allows us to test whether monotonicity violations are driven primarily by cross-crime heterogeneity in prosecutorial preferences or by within-crime disagreement.

Intervention

This study is a vignette measurement experiment. There is no behavioral intervention. Participating prosecutors complete a short online survey hosted on Qualtrics in which they read four written criminal-case vignettes and, for each case, recommend a sentence type (unsuspended imprisonment, suspended prison sentence, restriction of liberty, or fine) and the corresponding severity (e.g., months of imprisonment, months of restriction of liberty, or the number and amount of daily fine rates). Suspended prison sentences additionally elicit the trial period and whether probation supervision is imposed. The primary purpose of the study is to measure how often prosecutors, deciding identical cases independently, disagree in a way that is inconsistent with a single stringency ranking (the pairwise crossing rate underlying the monotonicity assumption used in examiner-IV designs). Because each prosecutor has time for only four vignettes, the four case-mix conditions in the analysis sample cannot all be presented to every prosecutor. Prosecutors are therefore randomly allocated by the Qualtrics built-in randomizer at survey entry across these conditions to ensure prosecutor characteristics (stringency, experience, office level, district) are balanced across conditions — the random allocation serves as a balance device, not as a behavioral treatment. The first 40 entrants are routed to a separate condition (Arm 1) reserved for projects outside this study and not used here. Subsequent entrants are allocated with equal probability across four analysis conditions, which differ in case-mix homogeneity: Arm 2 — four shoplifting cases (homogeneous); Arm 3 — four property-crime cases (shoplifting, embezzlement, burglary); Arm 4 — a mixed set (DUI, burglary, narcotics, shoplifting); Arm 5 — four diverse crime types (fraud, narcotics, driving ban, shoplifting). One shoplifting vignette appears as a common anchor in all four analysis conditions. Within each condition, all prosecutors face the same four vignettes in the same order. The survey also collects gender, years of experience, office level, and district. Prosecutors decide independently and receive no feedback or monetary incentive tied to responses. At the end of the survey, participants may optionally provide an email to be contacted for a planned later wave; the present registration covers the first wave only.

Experimental design

The study is a vignette measurement experiment with up to 250 Polish public prosecutors. The unit of random allocation is the individual prosecutor. There is no behavioral intervention and no individual-level treatment effect is estimated: the primary estimand is a within-condition measure of prosecutor disagreement (the pairwise crossing rate), and the secondary estimand is a comparison of this measure across case-mix conditions. The AEA form's treatment-vs-control frame therefore does not map onto the primary analysis. Random allocation is used because each prosecutor has time for only four vignettes, so the four case-mix conditions cannot all be presented to every prosecutor; the allocation balances prosecutor characteristics across conditions but does not itself constitute a behavioral treatment. At survey entry each prosecutor is allocated by the Qualtrics built-in randomizer to one of five arms (Arms 1–5). Arm 1 (the first 40 entrants) is reserved for projects outside this study and is not analyzed here. Arms 2–5 (~52 prosecutors each, ~208 in total) constitute the analysis sample for this registration and differ in case-mix homogeneity: Arm 2 — four shoplifting cases (homogeneous); Arm 3 — four property-crime cases (shoplifting, embezzlement, burglary; somewhat homogeneous); Arm 4 — a mixed set (DUI, burglary, narcotics, shoplifting; heterogeneous); Arm 5 — four diverse crime types (fraud, narcotics, driving ban, shoplifting; most heterogeneous). One shoplifting vignette appears as a common anchor in all four of Arms 2–5. Within each arm, all prosecutors face the same four vignettes in the same order; there is no within-arm randomization. For the primary within-arm monotonicity test all four analysis arms are symmetric — no arm functions as treatment or control. For the secondary cross-arm contrast (the heterogeneity gap T = VR(4+5) − VR(2+3) and the per-arm gradient), Arms 4+5 (heterogeneous case mix) play the role of treatment and Arms 2+3 (homogeneous case mix) the role of control. For each vignette the prosecutor recommends a sentence type (unsuspended imprisonment, suspended prison, restriction of liberty, fine) and the corresponding severity. The survey collects basic background characteristics. Prosecutors decide independently and do not interact.

Design details

Randomization method
Computer-generated random allocation within Qualtrics at survey entry, using the Qualtrics built-in randomizer. The allocation serves as a balance device to distribute prosecutor characteristics across the four case-mix conditions, since each prosecutor has time for only four vignettes and cannot see every condition; it does not assign a behavioral treatment. The first 40 entrants are routed deterministically to Arm 1 (reserved for projects outside this study). Each subsequent entrant is allocated with equal probability to one of Arms 2–5 until each analysis arm reaches its target of ~52 prosecutors.
Randomization unit
Individual prosecutor.
Clusters
250 prosecutors (no clustering; cluster equals unit of observation).
Observations
250 prosecutors providing 4 vignette decisions each (1,000 prosecutor-vignette observations). The monotonicity analysis uses Groups 2–5 (~208 prosecutors, ~832 prosecutor-vignette observations).
Arms
Caveat: this study has no behavioral treatment or control in the standard RCT sense — the figures below describe random allocation across four case-mix conditions used to measure within-condition prosecutor disagreement. Arm 1 (reserved for projects outside this study; not analyzed in the monotonicity test): 40 prosecutors. Arm 2 — four shoplifting cases (homogeneous case mix; cross-arm control role): ~52 prosecutors. Arm 3 — four property-crime cases (homogeneous case mix; cross-arm control role): ~52 prosecutors. Arm 4 — mixed crime types (heterogeneous case mix; cross-arm treatment role): ~52 prosecutors. Arm 5 — four diverse crime types (heterogeneous case mix; cross-arm treatment role): ~52 prosecutors. Total: up to 250 prosecutors. For the within-arm primary monotonicity test all four analysis arms (Arms 2–5) are symmetric and no arm functions as treatment or control; for the secondary heterogeneity-gradient test, Arms 2+3 pooled serve as the homogeneous (control) arm and Arms 4+5 pooled as the heterogeneous (treatment) arm.
Primary outcomes
The primary endpoint is the ranking-free pairwise crossing rate of prosecutor incarceration decisions, computed for two outcome definitions: (i) a binary incarceration indicator (1 if the prosecutor recommends unsuspended imprisonment for the case, 0 otherwise) and (ii) imprisonment length in months (months of unsuspended imprisonment, with non-imprisonment sentences coded as zero).
Secondary outcomes
(1) Case-level Imbens–Angrist monotonicity violation rate. (2) Average monotonicity violation rate and the implied 2SLS bias under a pre-specified severe-heterogeneity DGP. (3) Sum of negative 2SLS weights across cases. (4) Case-level own-weights and absolute cross-weights from the multi-treatment 2SLS decomposition at three pre-specified partitions of the sentence-type space (two, three, and four categories). (5) Alternative monotonicity violation rates corresponding to non-LATE estimands: extreme-pair monotonicity (LATE between extreme prosecutors), lenient-prosecutor monotonicity (LATT), and stringent-prosecutor monotonicity (LATUT). (6) Pooled-arm heterogeneity gap T = VR(4+5) − VR(2+3) comparing the heterogeneous arm to the homogeneous arm, and the per-group gradient VR_g across g ∈ {2,3,4,5}. (7) Subsampling pattern: violation rates by simulated group size m ∈ {2,3,5,9,15,N_g} and tercile of within-subsample stringency standard deviation.
Sponsors
Name: Czech Science Foundation Location: Prague, CZ Url: https://gacr.cz/en/ Name: Center for Inequality and Open Society Location: Prague, CZ Url: https://cios.law.prf.cuni.cz/

Cited by

No citation data available for this registration yet.

Keywords