Operator Gender Effects and Natural Language Processing Calibration in Violence Against Women Emergency Response: A Randomized Controlled Trial in Bogotá, Colombia

How to cite

Arias, Natalia Galvis. 2026. "Operator Gender Effects and Natural Language Processing Calibration in Violence Against Women Emergency Response: A Randomized Controlled Trial in Bogotá, Colombia." AEA RCT Registry. AEARCTR-0017873. https://doi.org/10.1257/rct.17873-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Natalia Galvis Arias*

* Primary investigator

Registry ID
AEARCTR-0017873
DOI
https://doi.org/10.1257/rct.17873-1.0
First registered
Last updated
Status
In development
Trial dates
2026-03-01 – 2026-06-30
Countries
Colombia (Bogotá, DC.)
JEL codes
C93, J16, O33, K42
Registry record
http://www.socialscienceregistry.org/trials/17873

Abstract

This study evaluates the impact of algorithmic decision support on emergency call classification and gender disparities in risk assessment at Bogotá’s Emergency Response Center (C4). I implement a 2×2 factorial randomized controlled trial crossing operator gender with randomized access to a machine-learning decision support system. A total of 275 operators across four shifts each classify approximately 20 archived, de-identified violence against women (VAW) call transcripts (5,440 observations).

Control operators follow standard classification procedures. Treatment operators receive real-time algorithmic risk suggestions, highlighted risk factors, and confidence intervals, while retaining full override authority. Calls are randomly assigned within risk strata, and expert consensus classifications from 80 specialists provide the benchmark.

I estimate three co-primary outcomes: (i) the average treatment effect of algorithmic assistance on accuracy relative to expert consensus and high-risk classification rates; (ii) gender differences in classification behavior; and (iii) whether algorithmic assistance mitigates gender gaps. The design is powered to detect a 6.2 percentage point effect on accuracy, a 4.5 percentage point gender gap in high-risk classification, and a 9.0 percentage point differential treatment effect by gender. The study provides experimental evidence on algorithmic decision support in high-stakes public safety contexts.

Intervention

We implement a 2×2 factorial RCT comparing algorithmic decision support versus standard manual procedures for classifying violence against women emergency calls, crossed with naturally occurring operator gender variation.

Control Condition: Operators classify archived call transcripts using existing C4 protocols—reviewing call information, conducting manual risk assessment, and assigning Low/Moderate/High risk classifications with written justification.

Treatment Condition: Operators perform identical classification task with real-time algorithmic assistance displaying: (1) predicted risk category with confidence score, (2) automatically extracted risk factors from call text, (3) linguistic pattern alerts (negation detection, hedging). Operators retain full override authority.

Assignment Mechanism: Stratified block randomization at operator level (strata: experience, shift, education). Each operator classifies ~20 randomly assigned archived calls. Random call assignment within operator eliminates selection bias from differential call exposure.

Training: All participants receive standardized 1.5-hour protocol: 1 hour VAW risk assessment principles, 30 minutes platform instruction. No algorithm-specific training to avoid differential treatment effects.

Sample: 275 operators from C4 workforce of 380, generating 5,440 total classifications over 2-month intervention.

Validation: Independent expert panels (16 panels, 80 Comisarías de Familia specialists) blindly classify random call subsample to establish accuracy benchmark. Experts blinded to operator characteristics and treatment status.

Experimental design

Design: 2×2 factorial randomized controlled trial crossing operator gender (naturally occurring) with algorithmic assistance (randomized).

Setting: Bogotá Emergency Response Center (C4), 2-month intervention (April-May 2026).

Sample: 275 operators across 4 shifts, 5,440 call classifications (~20 per operator).

Randomization: Individual operators randomized to algorithm-assisted or standard protocol, stratified by shift. Four 6-hour shifts serve as randomization strata:

Shift 1: 5:45 AM - 11:45 AM
Shift 2: 11:45 AM - 5:45 PM
Shift 3: 5:45 PM - 9:45 PM
Shift 4: 9:45 PM - 5:45 AM

Call Assignment: Each operator receives ~20 archived, de-identified calls randomly assigned within risk-level strata.
Interventions:

Control: Standard manual classification
Treatment: Algorithmic decision support with full operator override authority

Training: Universal 1.5-hour protocol (1 hour VAW risk assessment, 30 minutes platform instruction).

Primary Outcomes:
1. Algorithm Treatment Effect: Classification accuracy vs. expert consensus; high-risk classification rate
2. Operator Gender Effect: Gender differences in high-risk classification and accuracy
3. Interaction Effect: Differential algorithm impact by gender (bias mitigation)

Identification:
1. Algorithm effect: Experimental variation via randomization
2. Gender effect: Conditional independence assumption leveraging random call assignment and rich controls
3. Interaction: Factorial design tests bias correction mechanism

Power: 275 operators, 5,440 observations, ICC=0.12, design effect=3.28 provides:
90% power for 6.2pp algorithm effect
85% power for 4.5pp gender effect
80% power for 9.0pp interaction effect

Analysis: Linear probability models with shift-clustered standard errors:

Design details

Randomization method
Computer-generated random assignment using stratified block randomization with cryptographically secure random number generator. Randomization occurs at operator level after informed consent and baseline assessment completion, with allocation concealment maintained through server-side sequence generation inaccessible to research personnel. Technical Implementation: Stratified by shift assignment (4 shifts) to ensure balance within shifts Block size 4 within strata ensuring local balance 1:1 allocation ratio (approximately 137-138 operators per arm) Fixed random seed for reproducibility Allocation sequence generated and stored on secure server prior to enrollment Research staff blinded to upcoming assignments during recruitment and enrollment
Randomization unit
Individual operators Operators randomized to algorithm-assisted or standard protocol. Randomization stratified by shift assignment (4 shifts) to ensure balance within shifts. Cluster. Treatment assigned at operator level. Multiple calls nested within each operator creates clustering. Standard errors clustered at shift level to account for within-shift correlation.
Clusters
4 work shifts (for clustering of standard errors) Four 6-hour shifts serve as clusters for statistical inference, though randomization occurs at operator level.
Observations
5,440 call classifications 275 operators × ~20 calls per operator = 5,440 total observations
Arms
Treatment Assignment: Control arm (standard protocol): ~137-138 operators, ~2,720 classifications Treatment arm (algorithm-assisted): ~137-138 operators, ~2,720 classifications Distribution Across Shifts (each shift balanced): Shift 1: ~35 treatment, ~34 control Shift 2: ~35 treatment, ~34 control Shift 3: ~35 treatment, ~34 control Shift 4: ~34 treatment, ~34 control
Primary outcomes
1. Algorithm Treatment Effect: The causal impact of algorithmic decision support on classification accuracy and risk assessment patterns. 2. Operator Gender Effect: The causal relationship between operator gender and classification decisions, conditional on observables. 3. Interaction Effect: Whether algorithmic assistance differentially affects male versus female operators' classification patterns (bias mitigation)
Secondary outcomes
Algorithm Performance (Treatment Arm): 1. Algorithm standalone accuracy vs. expert consensus 2. Precision and recall for high-risk classification 3. F1-score and AUC-ROC 4. Calibration (Brier score) 5. Fairness metrics (equalized odds, demographic parity) Operator-Algorithm Interaction (Treatment Arm): 6. Override rate (% disagreements with algorithm) 7. Override accuracy 8. Agreement by algorithm confidence levels Heterogeneous Treatment Effects: 9. Algorithm effect by operator experience 10. Gender effect by call ambiguity 11. Algorithm effect by shift
Sponsors
Name: University of Manchester Location: Manchester, United Kingdom Url: https://www.manchester.ac.uk

Cited by

No citation data available for this registration yet.

Keywords