Improving procurement practices is crucial for increasing government efficiency because public procurement consumes 12 per cent of the global GDP (Bosio et al., 2022). Moreover, governments increasingly adopt digital platforms to procure goods and services, generating vast datasets as a byproduct. Despite the increasing adoption of early warning alarms in public procurement, whether and how these artificial intelligence tools increase government efficiency remains an open question. Using the Brazilian public sector as a laboratory, we randomly allocate messages reporting the presence of indications of passive waste in procurement documents to managers and quantify their effect on procurement outcomes.
Intervention
Intervention. We evaluate the impact of sending information revealing textual patterns in procurement documents that indicate the presence of passive waste in public procurement in the Brazilian public sector. More precisely, within a sample of purchase acts with at least one early warning alarm, we randomly allocate e-mails to the purchase acts' managers that reveal textual patterns suggesting: - Opportunities for efficiency gains (treatment 1). - Unnecessary restrictions to competition (treatment 2).
Experimental design
Experimental design. We randomly allocate e-mails to public managers revealing early warning alarms based of textual patterns in procurement documents that suggest the presence of passive waste. We divide the early warning alarms in two groups: one suggesting opportunities for efficiency gains (treatment 1) and another indicating unnecessary competition restrictions (treatment 2).
We used a clustered randomization design at the purchase act level. In the Brazilian procurement system, documents regulating the procurement auctions are published at the purchase act level, which are groups of procurement auctions managed by the same organization. In this case, by construction, the textual patterns suggesting the presence of passive waste affect all auctions within a purchase act. Therefore, while our unit of analysis is the procurement auction, our two treatments are randomized at the purchase act level.
Our population of interest is the set of auctions belonging to purchase acts with at least one early warning alarm during the two years of impact evaluation. We expect a sample with 257 purchase acts (number of clusters) and 822 auctions (number of observations).
Design details
Randomization method
Randomization method. For legal and logistical reasons, the Brazilian Court of Audits (TCU) randomizes the treatment allocation and sends the messages to our receiving organization (DNIT). The randomization process happens in four steps. First, TCU's artificial intelligence tools select each purchase act with at least one early-warning alarm. Second, the software Microsoft Power BI generates a data entry identified by the pair (purchase act ID, type of alarm), uniquely identifying each group of early earnings alarms within each purchase acts. Third, the software allocates each entry to control and treatment groups and prepares an automated e-mail for each distinct alarm in the data entries assigned to the treatment group. Fourth, for each purchase act, the chief of DNIT's internal control receives the automated e-mails and forwards them to the local office manager implementing the purchase act.
We describe a didactical example to increase our readers' understanding. Assume that the purchase act with identifier PA1 has five early warning alarms belonging to two groups: two text patterns suggesting restriction to competition (RC) and three indicating opportunities for efficiency gains (EG). In this case, the software generates two data entries - i.e., PA1-RC and PA1-EG - and two randomizations. If we assume that the software randomly allocates PA1-RC to the treatment group and PA1-EG to the control group, the manager of the purchase act receives two e-mails, one for each early warning alarm suggesting restriction to competition (RC).
Randomization unit
Randomization unit. Our unit of randomization is the unique identifier of a group of alarms within a purchase act, implying that we are randomizing our two treatments at the purchase act level.
Clusters
Planned number of clusters. We estimate the number of clusters based on monthly frequencies of early warning alarms among DNIT's purchase acts in the thirty-six months between July 2019 and June 2022. The average number of purchase acts with at least one early warning alarm per month and non-missing data for our primary outcomes is 10.72 in the entire sample. The same averages are 3.52 in the sample of purchase acts with at least one alarm suggesting unnecessary restrictions to competition (sample 1) and 8.16 in the one with at least one alarm suggesting opportunities for efficiency gains (sample 2). Therefore, for an experimental evaluation of two years of duration, we expect 257 purchase acts in the entire sample, 85 in sample 1, and 196 in sample 2.
Such projections are conservative because they still do not include purchase acts in the Regime Diferenciado de Contratação (RDC), a procurement modality created to accelerate selected infrastructure projects.
Observations
Planned number of observations. Given that we estimate an average of 3.2 auctions per purchase acts without missing variable for the outcomes in our sample, the estimates about the number of clusters imply an expected sample size of 822 auctions in the full sample, 272 in sample 1, and 627 in sample 2.
Arms
Sample size (or the number of clusters) by treatment arms. As we divide the purchase acts evenly among control and treatment groups, we expect 129 purchase acts in at least one of the treatments, 43 purchase acts in treatment 2, and 98 purchase acts in treatment 2.
Primary outcomes
Primary outcomes. We will estimate the effect of our two treatments on three primary procurement outcomes. First, to compute the percentage price change caused by our treatments, we will use the logarithm of the auction price as an outcome. Second, we will estimate treatment effects on the auction rebate, which measures the percentage discount of the auction concerning its reference price. Third, we estimate treatment effects on value for money measures, which compares the prices in our sample with average prices of the same goods outside our sample.
Secondary outcomes
Secondary outcomes. We estimate treatment effects on the quality of the goods and auction competitiveness. If we can merge the procurement and contract management phases accurately, we plan to include time- and cost- overrun measures in the analysis.
Cited by
No citation data available for this registration yet.