Testing Identifying Assumptions in Examiner IV Designs: A Vignette Experiment with Polish Prosecutors
How to cite
Sigstad, Henrik; Šoltés, Michal; Uhl, Andrzej. 2026. "Testing Identifying Assumptions in Examiner IV Designs: A Vignette Experiment with Polish Prosecutors." AEA RCT Registry. AEARCTR-0018783. https://doi.org/10.1257/rct.18783-1.1
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Many empirical studies in economics, criminology, and public policy use the random assignment of cases to decision-makers—judges, prosecutors, bail magistrates, disability examiners, patent reviewers, and caseworkers—to identify the causal effects of their decisions. These “examiner IV” designs rely on a monotonicity assumption: examiners who are stricter on average must be weakly stricter for every individual case. Yet this assumption is essentially untestable in administrative data, because the same case is rarely decided by multiple examiners.
This study conducts a vignette experiment with up to 250 Polish public prosecutors, each of whom independently recommends a sentence, including both type and severity, for the same fixed set of criminal cases. Observing many prosecutors’ decisions on identical cases allows us to measure how often monotonicity fails when decisions are made independently, without the panel-deliberation dynamics that complicate prior evidence from judicial panels. It also allows us to compute directly the case-level weights that determine whether the standard practice of controlling for non-focal decision propensities yields a well-defined estimand when examiners make multidimensional choices.
Prosecutors are randomized into four groups that vary in the homogeneity of their case mix, ranging from all-shoplifting cases to a diverse set of crime types. This design allows us to test whether monotonicity violations are driven primarily by cross-crime heterogeneity in prosecutorial preferences or by within-crime disagreement.
Intervention
This study is a vignette measurement experiment. There is no behavioral intervention. Participating prosecutors complete a short online survey hosted on Qualtrics in which they read four written criminal-case vignettes and, for each case, recommend a sentence type (unsuspended imprisonment, suspended prison sentence, restriction of liberty, or fine) and the corresponding severity (e.g., months of imprisonment, months of restriction of liberty, or the number and amount of daily fine rates). Suspended prison sentences additionally elicit the trial period and whether probation supervision is imposed. The primary purpose of the study is to measure how often prosecutors, deciding identical cases independently, disagree in a way that is inconsistent with a single stringency ranking (the pairwise crossing rate underlying the monotonicity assumption used in examiner-IV designs). Because each prosecutor has time for only four vignettes, the four case-mix conditions in the analysis sample cannot all be presented to every prosecutor. Prosecutors are therefore randomly allocated by the Qualtrics built-in randomizer at survey entry across these conditions to ensure prosecutor characteristics (stringency, experience, office level, district) are balanced across conditions — the random allocation serves as a balance device, not as a behavioral treatment. The first 40 entrants are routed to a separate condition (Arm 1) reserved for projects outside this study and not used here. Subsequent entrants are allocated with equal probability across four analysis conditions, which differ in case-mix homogeneity: Arm 2 — four shoplifting cases (homogeneous); Arm 3 — four property-crime cases (shoplifting, embezzlement, burglary); Arm 4 — a mixed set (DUI, burglary, narcotics, shoplifting); Arm 5 — four diverse crime types (fraud, narcotics, driving ban, shoplifting). One shoplifting vignette appears as a common anchor in all four analysis conditions. Within each condition, all prosecutors face the same four vignettes in the same order. The survey also collects gender, years of experience, office level, and district. Prosecutors decide independently and receive no feedback or monetary incentive tied to responses. At the end of the survey, participants may optionally provide an email to be contacted for a planned later wave; the present registration covers the first wave only.
Experimental design
The study is a vignette measurement experiment with up to 250 Polish public prosecutors. The unit of random allocation is the individual prosecutor. There is no behavioral intervention and no individual-level treatment effect is estimated: the primary estimand is a within-condition measure of prosecutor disagreement (the pairwise crossing rate), and the secondary estimand is a comparison of this measure across case-mix conditions. The AEA form's treatment-vs-control frame therefore does not map onto the primary analysis. Random allocation is used because each prosecutor has time for only four vignettes, so the four case-mix conditions cannot all be presented to every prosecutor; the allocation balances prosecutor characteristics across conditions but does not itself constitute a behavioral treatment. At survey entry each prosecutor is allocated by the Qualtrics built-in randomizer to one of five arms (Arms 1–5). Arm 1 (the first 40 entrants) is reserved for projects outside this study and is not analyzed here. Arms 2–5 (~52 prosecutors each, ~208 in total) constitute the analysis sample for this registration and differ in case-mix homogeneity: Arm 2 — four shoplifting cases (homogeneous); Arm 3 — four property-crime cases (shoplifting, embezzlement, burglary; somewhat homogeneous); Arm 4 — a mixed set (DUI, burglary, narcotics, shoplifting; heterogeneous); Arm 5 — four diverse crime types (fraud, narcotics, driving ban, shoplifting; most heterogeneous). One shoplifting vignette appears as a common anchor in all four of Arms 2–5. Within each arm, all prosecutors face the same four vignettes in the same order; there is no within-arm randomization. For the primary within-arm monotonicity test all four analysis arms are symmetric — no arm functions as treatment or control. For the secondary cross-arm contrast (the heterogeneity gap T = VR(4+5) − VR(2+3) and the per-arm gradient), Arms 4+5 (heterogeneous case mix) play the role of treatment and Arms 2+3 (homogeneous case mix) the role of control. For each vignette the prosecutor recommends a sentence type (unsuspended imprisonment, suspended prison, restriction of liberty, fine) and the corresponding severity. The survey collects basic background characteristics. Prosecutors decide independently and do not interact.
Design details
Cited by
No citation data available for this registration yet.