Strengthening Accountability Chains: Measuring Impact in Health Service Delivery in Uganda
How to cite
Grepin, Karen; Atuyambe, Lynn; Jack, William; Habyarimana, James. 2015. "Strengthening Accountability Chains: Measuring Impact in Health Service Delivery in Uganda." AEA RCT Registry. AEARCTR-0000798. https://doi.org/10.1257/rct.798-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Increasing transparency and accountability in the heath sector is thought to improve health service delivery outcomes; however, there is little empirical evidence on the ways in which this might occur in a developing country setting. In Uganda, UNICEF in partnership with the Ministry of Health of Uganda has developed facility-level performance indicator reports (PIRs) that are based on administrative data and can be distributed to health facilities via mTrac - an innovative mobile phone platform that is operating nationally at scale. The PIRs provide health facilities with detailed information on their performance on four key reproductive, maternal, newborn and child health indicators relative to other facilities and relative to their past performance. Using a randomized control experimental design and targeting approximately 1400 facilities, this project aims to test the impact of increased transparency through the sharing of performance indicator reports with front line health workers on facility-level performance after 12 months. Simultaneously, we plan to evaluate the impact of also sharing similar performance reports with district health officials in a random sample of about half of 95 districts in the country. The goal is to shed light on the ways in which strengthening accountability chains in Uganda might help improve health system performance.
Intervention
The project consists of two interventions implemented simultaneously. Using a factorial design, each intervention will be measured independently in addition to the interaction effect between interventions.
The first intervention are facility level PIRs sent to health facilities monthly via mobile phone. The reports use Ministry of Health (MoH) Health Management Information System (HMIS) data and compare performance over the past 3 months to performance at the same time 12 months before. A PIR comprises 3 SMS reports. The three different SMS reports sent each month are described below:
1. AVERAGE THREE MONTH REPORT: The average of each indicator for the previous three months. The indicators are first antenatal care visit, fourth antenatal care visit, deliveries and third doses of PCV vaccine administered. In addition, the reporting rate for the past three months will be revealed. If a facility fails to report an indicator in a given month, the score for that indicator in that month is zero.
3. CHANGE FROM LAST YEAR: The average three month report from this year compared to the same period last year. For example, the August report from this year is composed of an average of May, June and July 2015, compared to May, June and July 2014. The total score represents overall change from last year and is an average of the individual changes observed among the four indicators.
3. RANK: A facility’s improvement compared to other facilities in their district. The biggest improvements from the previous year rank highest. HCIIs and HCIIIs are ranked separately, so facilities are told their rank out of the number of HCII or HCIII facilities in their district. The report also names the facility performing just above them and the facility performing just below.
The second intervention is district level PIRs sent to the District Health Office every month. One report is sent by SMS, and a second report by email. The components of these message are as follows:
1. EMAIL- The district health office receives the same monthly reports as the facility. The three reports are aggregated into a single line on a spreadsheet into an email that will be sent from the MoH. The email will contain the same data as the three reports shown above so that the District Health Office (DHO) can follow-up with the facilities that are under-performing and congratulate the leaders.
2. SMS - At the same time, an SMS will be sent to the DHO identifying the top-3 and bottom-3 performing facilities in a district as well as a reminder to the DHO to look at the full facility level reports, which have been sent by email.
Experimental design
Our randomization strategy will take place at the facility and district levels using a factorial design. First, the 112 districts in Uganda were analyzed for existing transparency and accountability initiatives. All districts without a major initiative underway were included in the study, which left 95 districts. This provides us with a sample of 1417 HCII and HCIII facilities.
The process of creating, disseminating and studying the impact of performance indicator reports is detailed below.
a. Facilities submit a monthly outpatient report on paper called HMIS 105. This report is sent to a technical officer who enters the data into DHIS2.
b. We downloaded data for every Health Center Level II and III in 95 districts, at the 1,417 health facilities. From this data, we will generate performance indicator reports for each facility selected to participate. These reports will be based on four key indicators:
i. Number of 1st antenatal care visits, number of 4th antenatal care visits, number of deliveries and number of PCV3 vaccines administered.
c. The 1,417 facilities will first be divided into two randomly selected groups, 708 treatment and 709 control facilities. Then the 95 districts will be divided into treatment and control districts:
i. In Treatment districts, District Health Teams will receive summary reports by mTrac SMS messages monthly and in email form monthly as well. The report sent to the district will be supplemented with a ranking of facilities, highlighting top-performers. Note that all facilities in treatment districts are included in reports.
ii. In control districts, no report will be sent to the District Health Teams. Note that these control districts will contain facilities that have been sent facility indicator reports directly.
d. The facility level treatment consists of facility level indicator reports sent directly to directors or “in-charge” and data officers at selected facilities by SMS through the mTrac system.
i.The report consists of a mean score of each chosen indicator as well as a measure of the frequency of reporting. The period of the report will cover a three month span to account for expected month to month fluctuations in service provision and the failure to report in a given month. Note that the report will be distributed monthly but is composed of the previous three months of data reported. In addition, the report will contain a comparison to performance during the same three month period 12-months before. Each indicator will be compared as well as an aggregate measure of change from the previous year. Last the change from last year will be used to rank facilities of the same type within a given district. The facility receiving the report will learn where they stand in the district in addition to receiving the name of the facility immediately above and below their position.
e.Note that treatment facilities in this arm will be contained in districts in which district level health officials receive information about facilities (treatment districts) and also in districts where level health officials do not receive information about facilities (control districts). Similarly, control facilities that themselves do not receive performance indicator reports will be contained within some districts that receive reports about that facility. Purely control facilities will be facilities that do not receive performance indicator reports and are located within control districts.
f. For our analysis, we plan to study additional outcomes not contained in the performance indicator reports such as reporting rates, number of outpatient attendees, malaria treatment rates and stock of tracer medications, all of which will also be sourced from the HMIS system.
i. To ensure reliability of the HMIS data, we will conduct facility level audits on a randomly selected subsample of facilities towards the end of the project. We estimate that 30% of facilities, roughly 100 facilities per treatment arm, will be audited. Our enumerators will ask the in-charge at the facility to view the facility’s records and record the data in addition to observing the condition of the facility at the time of the visit.
Design details
Cited by
No citation data available for this registration yet.