Strengthening Accountability Chains: Measuring Impact in Health Service Delivery in Uganda

How to cite

Grepin, Karen; Atuyambe, Lynn; Jack, William; Habyarimana, James. 2015. "Strengthening Accountability Chains: Measuring Impact in Health Service Delivery in Uganda." AEA RCT Registry. AEARCTR-0000798. https://doi.org/10.1257/rct.798-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Karen Grepin* , Lynn Atuyambe , William Jack , James Habyarimana

* Primary investigator

Registry ID
AEARCTR-0000798
DOI
https://doi.org/10.1257/rct.798-1.0
First registered
Last updated
Status
In development
Trial dates
2015-10-21 – 2016-12-31
Countries
Private
JEL codes
I18, H11, L32
Registry record
http://www.socialscienceregistry.org/trials/798

Abstract

Increasing transparency and accountability in the heath sector is thought to improve health service delivery outcomes; however, there is little empirical evidence on the ways in which this might occur in a developing country setting. In Uganda, UNICEF in partnership with the Ministry of Health of Uganda has developed facility-level performance indicator reports (PIRs) that are based on administrative data and can be distributed to health facilities via mTrac - an innovative mobile phone platform that is operating nationally at scale. The PIRs provide health facilities with detailed information on their performance on four key reproductive, maternal, newborn and child health indicators relative to other facilities and relative to their past performance. Using a randomized control experimental design and targeting approximately 1400 facilities, this project aims to test the impact of increased transparency through the sharing of performance indicator reports with front line health workers on facility-level performance after 12 months. Simultaneously, we plan to evaluate the impact of also sharing similar performance reports with district health officials in a random sample of about half of 95 districts in the country. The goal is to shed light on the ways in which strengthening accountability chains in Uganda might help improve health system performance.

Intervention

The project consists of two interventions implemented simultaneously. Using a factorial design, each intervention will be measured independently in addition to the interaction effect between interventions.

The first intervention are facility level PIRs sent to health facilities monthly via mobile phone. The reports use Ministry of Health (MoH) Health Management Information System (HMIS) data and compare performance over the past 3 months to performance at the same time 12 months before. A PIR comprises 3 SMS reports. The three different SMS reports sent each month are described below:

1. AVERAGE THREE MONTH REPORT: The average of each indicator for the previous three months. The indicators are first antenatal care visit, fourth antenatal care visit, deliveries and third doses of PCV vaccine administered. In addition, the reporting rate for the past three months will be revealed. If a facility fails to report an indicator in a given month, the score for that indicator in that month is zero.

3. CHANGE FROM LAST YEAR: The average three month report from this year compared to the same period last year. For example, the August report from this year is composed of an average of May, June and July 2015, compared to May, June and July 2014. The total score represents overall change from last year and is an average of the individual changes observed among the four indicators.

3. RANK: A facility’s improvement compared to other facilities in their district. The biggest improvements from the previous year rank highest. HCIIs and HCIIIs are ranked separately, so facilities are told their rank out of the number of HCII or HCIII facilities in their district. The report also names the facility performing just above them and the facility performing just below.

The second intervention is district level PIRs sent to the District Health Office every month. One report is sent by SMS, and a second report by email. The components of these message are as follows:

1. EMAIL- The district health office receives the same monthly reports as the facility. The three reports are aggregated into a single line on a spreadsheet into an email that will be sent from the MoH. The email will contain the same data as the three reports shown above so that the District Health Office (DHO) can follow-up with the facilities that are under-performing and congratulate the leaders.

2. SMS - At the same time, an SMS will be sent to the DHO identifying the top-3 and bottom-3 performing facilities in a district as well as a reminder to the DHO to look at the full facility level reports, which have been sent by email.

Experimental design

Our randomization strategy will take place at the facility and district levels using a factorial design. First, the 112 districts in Uganda were analyzed for existing transparency and accountability initiatives. All districts without a major initiative underway were included in the study, which left 95 districts. This provides us with a sample of 1417 HCII and HCIII facilities.

The process of creating, disseminating and studying the impact of performance indicator reports is detailed below.

a. Facilities submit a monthly outpatient report on paper called HMIS 105. This report is sent to a technical officer who enters the data into DHIS2.
b. We downloaded data for every Health Center Level II and III in 95 districts, at the 1,417 health facilities. From this data, we will generate performance indicator reports for each facility selected to participate. These reports will be based on four key indicators:
i. Number of 1st antenatal care visits, number of 4th antenatal care visits, number of deliveries and number of PCV3 vaccines administered.
c. The 1,417 facilities will first be divided into two randomly selected groups, 708 treatment and 709 control facilities. Then the 95 districts will be divided into treatment and control districts:
i. In Treatment districts, District Health Teams will receive summary reports by mTrac SMS messages monthly and in email form monthly as well. The report sent to the district will be supplemented with a ranking of facilities, highlighting top-performers. Note that all facilities in treatment districts are included in reports.
ii. In control districts, no report will be sent to the District Health Teams. Note that these control districts will contain facilities that have been sent facility indicator reports directly.

d. The facility level treatment consists of facility level indicator reports sent directly to directors or “in-charge” and data officers at selected facilities by SMS through the mTrac system.
i.The report consists of a mean score of each chosen indicator as well as a measure of the frequency of reporting. The period of the report will cover a three month span to account for expected month to month fluctuations in service provision and the failure to report in a given month. Note that the report will be distributed monthly but is composed of the previous three months of data reported. In addition, the report will contain a comparison to performance during the same three month period 12-months before. Each indicator will be compared as well as an aggregate measure of change from the previous year. Last the change from last year will be used to rank facilities of the same type within a given district. The facility receiving the report will learn where they stand in the district in addition to receiving the name of the facility immediately above and below their position.
e.Note that treatment facilities in this arm will be contained in districts in which district level health officials receive information about facilities (treatment districts) and also in districts where level health officials do not receive information about facilities (control districts). Similarly, control facilities that themselves do not receive performance indicator reports will be contained within some districts that receive reports about that facility. Purely control facilities will be facilities that do not receive performance indicator reports and are located within control districts.
f. For our analysis, we plan to study additional outcomes not contained in the performance indicator reports such as reporting rates, number of outpatient attendees, malaria treatment rates and stock of tracer medications, all of which will also be sourced from the HMIS system.
i. To ensure reliability of the HMIS data, we will conduct facility level audits on a randomly selected subsample of facilities towards the end of the project. We estimate that 30% of facilities, roughly 100 facilities per treatment arm, will be audited. Our enumerators will ask the in-charge at the facility to view the facility’s records and record the data in addition to observing the condition of the facility at the time of the visit.

Design details

Randomization method
Facilities and districts were randomized using the "randomize" command in Stata 13.0. Districts were balanced on their population and proportion of the population defined as "rural" by the 2014 Uganda Census. Districts were then blocked by region. Facilities were balanced on the average number of reported antenatal care first and fourth visits as well as number of deliveries. Historical data for the entire 12 month period used for the randomization did not exist for pneumococcal conjugate vaccine and this variable was omitted for the purpose of balance. Facilities were then blocked by their type as classified by the MoH, II or III.
Randomization unit
The randomization unit is the health facility. The study is randomized at two levels. First, the 95 eligible districts in Uganda are randomly divided into treatment and control. Then the facilities within the districts are randomly divided into two groups.
Clusters
Not clustered
Observations
1417 health facilities.
Arms
Treatment Facilities in Control Districts: 374 Treatment Facilities in Treatment Districts: 334 Control Facilites in Control Districts: 374 Control Facilites in Treatment Districts: 335
Primary outcomes
The key outcome variables of interest are increases in the utilization of reproductive, maternal, neonatal, and child (RMNCH) health services (RMNCH) as measured by data in the HMIS system. Specifically, we will look for changes in the indicators included in the reports sent to facilities and districts; number of first antenatal care visits, the number of fourth antenatal care visits, number of deliveries performed at health facilities and the number of pneumococcal conjugate vaccine third doses delivered. In addition, we will also use actual reporting rates as an outcome variable. In addition to the outcomes in the HMIS system, we will conduct an endline facility survey to get a more comprehensive measure of overall health facility quality in a random sample of 30% of the health facilities in 50% of the districts. The endline survey will capture many dimensions of facility quality: facility operating hours, availability of key equipment or facilities, health financing, training, availability and stock out of commodities and medicines, data that can be corroborated in HMIS reports, etc. The facility visits will be unannounced.

Cited by

No citation data available for this registration yet.

Keywords