Operator Gender Effects and Natural Language Processing Calibration in Violence Against Women Emergency Response: A Randomized Controlled Trial in Bogotá, Colombia
How to cite
Arias, Natalia Galvis. 2026. "Operator Gender Effects and Natural Language Processing Calibration in Violence Against Women Emergency Response: A Randomized Controlled Trial in Bogotá, Colombia." AEA RCT Registry. AEARCTR-0017873. https://doi.org/10.1257/rct.17873-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This study evaluates the impact of algorithmic decision support on emergency call classification and gender disparities in risk assessment at Bogotá’s Emergency Response Center (C4). I implement a 2×2 factorial randomized controlled trial crossing operator gender with randomized access to a machine-learning decision support system. A total of 275 operators across four shifts each classify approximately 20 archived, de-identified violence against women (VAW) call transcripts (5,440 observations).
Control operators follow standard classification procedures. Treatment operators receive real-time algorithmic risk suggestions, highlighted risk factors, and confidence intervals, while retaining full override authority. Calls are randomly assigned within risk strata, and expert consensus classifications from 80 specialists provide the benchmark.
I estimate three co-primary outcomes: (i) the average treatment effect of algorithmic assistance on accuracy relative to expert consensus and high-risk classification rates; (ii) gender differences in classification behavior; and (iii) whether algorithmic assistance mitigates gender gaps. The design is powered to detect a 6.2 percentage point effect on accuracy, a 4.5 percentage point gender gap in high-risk classification, and a 9.0 percentage point differential treatment effect by gender. The study provides experimental evidence on algorithmic decision support in high-stakes public safety contexts.
Intervention
We implement a 2×2 factorial RCT comparing algorithmic decision support versus standard manual procedures for classifying violence against women emergency calls, crossed with naturally occurring operator gender variation.
Control Condition: Operators classify archived call transcripts using existing C4 protocols—reviewing call information, conducting manual risk assessment, and assigning Low/Moderate/High risk classifications with written justification.
Treatment Condition: Operators perform identical classification task with real-time algorithmic assistance displaying: (1) predicted risk category with confidence score, (2) automatically extracted risk factors from call text, (3) linguistic pattern alerts (negation detection, hedging). Operators retain full override authority.
Assignment Mechanism: Stratified block randomization at operator level (strata: experience, shift, education). Each operator classifies ~20 randomly assigned archived calls. Random call assignment within operator eliminates selection bias from differential call exposure.
Training: All participants receive standardized 1.5-hour protocol: 1 hour VAW risk assessment principles, 30 minutes platform instruction. No algorithm-specific training to avoid differential treatment effects.
Sample: 275 operators from C4 workforce of 380, generating 5,440 total classifications over 2-month intervention.
Validation: Independent expert panels (16 panels, 80 Comisarías de Familia specialists) blindly classify random call subsample to establish accuracy benchmark. Experts blinded to operator characteristics and treatment status.
Experimental design
Design: 2×2 factorial randomized controlled trial crossing operator gender (naturally occurring) with algorithmic assistance (randomized).
Setting: Bogotá Emergency Response Center (C4), 2-month intervention (April-May 2026).
Sample: 275 operators across 4 shifts, 5,440 call classifications (~20 per operator).
Randomization: Individual operators randomized to algorithm-assisted or standard protocol, stratified by shift. Four 6-hour shifts serve as randomization strata:
Shift 1: 5:45 AM - 11:45 AM
Shift 2: 11:45 AM - 5:45 PM
Shift 3: 5:45 PM - 9:45 PM
Shift 4: 9:45 PM - 5:45 AM
Call Assignment: Each operator receives ~20 archived, de-identified calls randomly assigned within risk-level strata.
Interventions:
Control: Standard manual classification
Treatment: Algorithmic decision support with full operator override authority
Training: Universal 1.5-hour protocol (1 hour VAW risk assessment, 30 minutes platform instruction).
Primary Outcomes:
1. Algorithm Treatment Effect: Classification accuracy vs. expert consensus; high-risk classification rate
2. Operator Gender Effect: Gender differences in high-risk classification and accuracy
3. Interaction Effect: Differential algorithm impact by gender (bias mitigation)
Identification:
1. Algorithm effect: Experimental variation via randomization
2. Gender effect: Conditional independence assumption leveraging random call assignment and rich controls
3. Interaction: Factorial design tests bias correction mechanism
Power: 275 operators, 5,440 observations, ICC=0.12, design effect=3.28 provides:
90% power for 6.2pp algorithm effect
85% power for 4.5pp gender effect
80% power for 9.0pp interaction effect
Analysis: Linear probability models with shift-clustered standard errors:
Design details
Cited by
No citation data available for this registration yet.