Selection Neglect in Policing Decisions

How to cite

Martini, Santiago De; Gonzalez, Juan Bautista; Perez-Vincent, Santiago María; Norza, Ervyn. 2025. "Selection Neglect in Policing Decisions." AEA RCT Registry. AEARCTR-0016438. https://doi.org/10.1257/rct.16438-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Santiago De Martini* , Juan Bautista Gonzalez , Santiago María Perez-Vincent , Ervyn Norza

* Primary investigator

Registry ID
AEARCTR-0016438
DOI
https://doi.org/10.1257/rct.16438-1.1
First registered
Last updated
Status
In development
Trial dates
2025-09-12 – 2025-09-20
Countries
Private
JEL codes
C91, J15
Registry record
http://www.socialscienceregistry.org/trials/16438

Abstract

In this paper, we design two framed field experiments to measure selection neglect and test whether it distorts policing decisions and beliefs about criminality, using a sample of high-ranking police officers from the National Police of Colombia. The experiment takes place in an abstract setting where participants are presented with hypothetical “neighborhoods,” each characterized by an unknown level of “crimes.” These neighborhoods belong to larger geographic units —cities— comprising multiple neighborhoods with varying crime levels, so neighborhoods are draws from the city distribution. In each round, participants see the number of reported crimes in each neighborhood, alongside the reporting rate of the neighborhood. In Experiment 1, participants are shown two neighborhoods each round, each from a different city, and are incentivized to patrol the area with the most crimes (reported and unreported). In Experiment 2, participants are shown only one neighborhood each round, always coming from the same city, and are asked to identify the mean number of crimes of that city. This design allows us to test whether participants properly account for the selection inherent in crime data—specifically, whether they adjust for the fact that reported crime depends on reporting rates. We examine whether in Experiment 1, differential reporting rates will bias patrolling decisions, and in Experiment 2, whether higher reporting rates lead to systematically higher crime estimates. Finally, we evaluate low-cost interventions aimed at mitigating selection neglect, with the goal of improving the efficiency and effectiveness of policing decisions in low- and middle-income countries.

Intervention

Experiment 1:

Experiment 1 consists of two blocks.

Block 1 is designed to estimate individual-level selection neglect and incorrect models. Each round, participants see information about two neighborhoods, each with a number of crimes drawn from two city-specific distributions (neighborhood A1 is a draw from city distribution A, neighborhood B1 is an independent draw from city distribution B). Importantly, the number of crimes is unobserved to participants, who instead see the following information:
- Number of reported crimes.
- Reporting rate (10-90%): average proportion of crimes that is reported. Drawn from uniform distribution [10,90] but only with "easy" numbers, to reduce cognitive requirements (multiples of 5 such as 10,20,25,35...).
- Average age (20-40): of the neighborhood residents.
- Unemployment rate (5-20%).
- Socioeconomic status (1-6).

Participants know they can infer the number of crimes just from the reporting rate and the reported crimes. All the other variables are independent draws from different distributions, so there's no relationship between these variables and crime. After seeing this information, participants choose which neighborhood to patrol, and are incentivized to patrol the neighborhood with the highest number of crimes (reported and unreported). After 9 choices, participants are informed they will only be able to see one piece of information apart from the number of reported crimes for the last decision, and asked to choose which variable they'd like to see. Then, they make a last decision with those two pieces of information.

This design allows us to identify (1) whether participants account for reported crime data selection, (2) which variables have more weight in their patrolling decisions, (3) which variables they identify as most informative of the number of crimes.

---
Block 2 uses a similar design but focuses on reported crimes and reporting rates. Now, participants see neighborhoods coming from 2 different cities, which have the same distribution of crimes (but participants don't know). In Block 2, participants only see the number of reported crimes and the reporting rate of each neighborhood, and make a patrolling decision based on that information. As in Block 1, they are incentivized to patrol the neighborhood with the highest number of crimes, and they make 15 decisions through the block. At the end of those 15 decisions, participants are incentivized to guess which city has the highest average number of crimes and by how much (up 5% more, between 5-15%, between 15-25%, more than 25% more).

We randomize participants into two groups for all the 15 decisions. For those in the Orthogonal group, the reporting rates of both neighborhoods are drawn from the same uniform distribution [10,90], as in Block 1. For those in the Differential group, the reporting rates of one city are drawn from a uniform [40,90], while those of the other city are drawn from [10,60]. This allows us to identify whether differential data selection generates statistical discrimination and overpolicing.

Additionally, for the last 7 decisions of the block we randomize participants into two groups. Those in the Exogenous group will see reporting rates drawn in the same way described above, keeping the Orthogonal/Differential treatment arm. For those in the Endogenous group, reporting rates will depend on their previous decision. We first draw all reporting rates in the same manner as for the exogenous group, but will add a "Reporting Bonus" of 15% to the next neighborhood of the city participants decide to patrol, and substract 15% from the next neighborhood of the city participants decide to not patrol. This mimics dynamics of data selection where patrolling generates more crime data (more police reports and arrests) which are then used to predict crime in the next turn.

Finally, for the last 7 decisions of the block we randomize participants into 3 groups. Those in the Algorithm group see a recommendation made by an algorithm that takes the current reported crimes data, adjusts it by reporting rate, and averages it with the average of the last 3 neighborhoods drawn from that city. The algorithm recommends to patrol the neighborhood with the highest average crime. Those in the Algorithm Neglect group see a recommendation from an algorithm that does exactly the same but without adjusting by reporting rates. Those in Control see no recommendation. Both algorithms use previous data, although this data is non-informative conditional on the current data, to mimic the functioning of hotspot and predictive policing algorithms. These interventions allow us to measure whether predictive policing algorithms can abate or reinforce cognitive biases.

At the end of Block 2, participants will make an additional patrolling decision, between neighborhoods coming from the same cities as in the rest of the block. Participants now choose while only seeing the number of reports. After this choice, they have the opportunity to buy the reporting rate of each neighborhood, for a randomly selected price. If they do, the price will only be discounted from their performance bonus in that decision if they make the correct choice. In other words, they will never lose money by buying. If they decide to "buy", they will repeat the decision with the new information. If they don't buy, Block 2 ends.

-----------------
Experiment 2

Block 1 is designed to estimate individual-level cognitive biases—particularly selection neglect. Participants engage in a belief updating task where they are shown a series of hypothetical neighborhoods within a city. Each neighborhood has a true (but unobserved) level of crime. Participants are told that crime levels across neighborhoods follow a normal distribution, and they are provided with the variance of that distribution.

Over the course of 15 rounds, participants receive information about one randomly drawn neighborhood per round. For each neighborhood, they observe (i) the number of reported crimes and (ii) the reporting rate—that is, the proportion of crimes that are reported to the police. Importantly, participants must recognize that the true number of crimes can be inferred by dividing the reported crimes by the reporting rate.

Using the variation in the information shown across rounds, we estimate two structural parameters for each participant:

1. Their Bayesian updating weight.
2. Their propensity to exhibit selection neglect—that is, the extent to which they fail to account for reporting rates when inferring crime levels.

In Block 2, participants are asked to compare two neighborhoods belonging to two different districts. Crime levels within each neighborhood are normally distributed, but the two districts differ in their means. Without seeing any specific neighborhood information in advance, participants must infer which neighborhood is likely to have the higher expected crime level and decide where to send a patrol. Their decision must therefore rely on information accumulated across previous rounds.

After making their patrol decision, participants are shown the number of reported crimes and the reporting rate for both neighborhoods. A key feature of this design is that the neighborhood receiving the patrol always has a highly precise reporting rate of 95%, while the other neighborhood has a less precise rate strictly below 95%. Every five rounds, participants are also asked to state their guess of the mean crime level for the neighborhood.

Experimental design

We conduct a framed field experiment with Colombian police officers to measure selection neglect and its effect on belief updating. In Block 1, participants observe noisy information on reported crimes and reporting rates across neighborhoods from a city and then to then make a guess on the average number of crimes that take place in the neighborhoods of a city. We use a Bayesian framework to estimate (1) the extent of selection neglect and (2) the relative weight placed on prior beliefs.

In Block 2, participants compare crime levels across two neighborhoods based on reported crime and reporting rates. We introduce two interventions: one where an intelligence agency provides a correct recommendation and another where it gives an incorrect one. We compare these to a control group to assess how external recommendations influence decision accuracy.

Design details

Randomization method
Randomization done in office by a computer
Randomization unit
Individual
Clusters
400 police officers were summoned by superior officers to participate in the study. Show-up is uncertain a priori. Additionally, participants might show-up to the summoning, but decide not to participate in the study.
Observations
Between 200 and 400 police officers.
Arms
Each arm will be (in expectation) of equal size and it will depend on the final number of participants. Because participants are summoned by superior officers to show up, we can not necessarily consider these participants as motivated. Although completion of the study will be incentivized, as well as correct performance, we anticipate that some participants will try to complete as fast as possible to get the completion bonus, while others will start and not continue. To ensure we obtain valuable data that reflects actual choices and not random behavior, in Experiment 2 (the most challenging, as it requires dynamic Bayesian updating), we will restrict the sample to those who make consistent predictions: their predictions should fall into the range between the minimum number of reports or adjusted reports (whichever is smaller) and the maximum number of reports or adjusted reports (whichever is higher). This exclusion criterion is to ensure we don't include responses that, for any level of selection neglect, are not consistent with the data participants are seeing, and most likely reflects random responses.
Primary outcomes
Experiment 1: Block 1 We will analyze how the probability of choosing A vs B depends on the exogenous variation of number of reported crimes, reporting rates, and the other decoy variables. We will characterize what "signal" participants perceive, this is, a convex combination of the number of reports and the rate-adjusted number of reports. We will also explore the uncertainty in participants decisions, measured in 3 ways: elicited directly, the number of choice switches each round before submitting, and response times. Finally, we will study which information source participants considered more valuable. Block 2 We will focus on characterizing which signal participants perceive, and constructing a measure of selection neglect from that. We will also explore uncertainty (as in block 1), and the effect of the treatments. We will explore whether differential selection rates generate statistical discrimination (through the belief elicitation) and over policing. ----------------- Experiment 2: In Block 1, we estimate two outcome variables at the individual level. The first is a parameter that captures the degree of selection neglect—that is, the extent to which a participant fails to account for the fact that observed crime data is shaped by civilian reporting behavior. A higher value of this parameter indicates a greater tendency to overlook the selective nature of reported crime data. The second is the individual's Bayesian updating weight (i.e., kallman gain), a standard parameter in the Bayesian learning literature. See attached pdf to learn more about the model. Importantly, we can recover both parameters using only participants’ crime-level guesses across rounds. Identification relies on the randomization of the information shown to participants in each round, which enables us to analyze how variation in observed data influences their predictions. In Block 2, we measure participants’ prediction accuracy, defined as the number of correct responses across rounds. We calculate accuracy separately for each treatment group and the control group to evaluate the impact of our interventions by comparing the number of correct answers in each treatment group.
Secondary outcomes
There will be variation also in the average reprting rate that each participant observes. In particular, we will be dividing participants into groups where the average reporting rate is higher than 100% and lower than 100%. Because of that, we expect that the participants in the group with a lower reporting rate are going to have lower guesses than the other group. This would be a manifestation of selection neglect because they are not using the reporting rate information to back up their beliefs on crime. We will compare the deviations from the true main crime rate across groups, as well as how close they get to the real mean crime rate of the city across rounds. For block 1 we will also compare the parameters estimated for different chunks of the data. This will be helpful because it can be the case that some participants learn about the selection process throughout the game and therefore register a much lower selection neglect for later rounds of the game. In block 2 we will measure the frequency of switches between neighborhoods. This will tell us about the value of exploration that agents put in gathering more precise information from the patrolls. For both experiments we register their response times, the number of times they use the calculator and click on the information of data of past rounds and the operations that they do on the computer.

Cited by

No citation data available for this registration yet.

Keywords