Data-Driven Instruction in Honduras: An Impact Evaluation of the EducAcción Promising Reading Intervention

How to cite

Liuzzi, Sarah; Toledo, Chantal; Glazerman, Steven; Murray, Nancy; Humpage-Liuzzi, Sarah. 2015. "Data-Driven Instruction in Honduras: An Impact Evaluation of the EducAcción Promising Reading Intervention." AEA RCT Registry. AEARCTR-0000780. https://doi.org/10.1257/rct.780-5.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Sarah Liuzzi* , Chantal Toledo , Steven Glazerman , Nancy Murray , Sarah Humpage-Liuzzi

* Primary investigator

Registry ID
AEARCTR-0000780
DOI
https://doi.org/10.1257/rct.780-5.0
First registered
Last updated
Status
On going
Trial dates
2014-10-01 – 2018-09-30
Countries
Private
JEL codes
I20
Registry record
http://www.socialscienceregistry.org/trials/780

Abstract

As policymakers’ focus in developing countries shifts from school access to school quality, student assessment has come to the fore. Governments use assessment data not only to monitor educational progress, but to provide feedback aimed at improving teaching and learning. In this evaluation, we focus on the potential effects of formative assessment (FA), in which teachers use frequent assessments of their students to improve their teaching, and end-of-grade (EOG) summative assessments, which, like FAs, can be the basis for teachers and principals to adapt their teaching practices and curricula to better meet the needs of their students.

We undertake a randomized controlled trial (RCT) in 180 schools in Honduras, complemented by qualitative analysis, to estimate the impacts of providing print materials and pedagogical support to maximize teachers’ ability to take advantage of these assessments. To do this, we randomize schools into 3 experimental groups: EOG wrap-around support and FA with support (Group A), EOG wrap-around support only (Group B), and ongoing programming by Honduras’ Ministry of Education (MOE) (Group C, the control group). Group C has limited access to printed FA materials from the Ministry’s previous efforts to distribute them. All schools administer an EOG test, but only schools in Groups A and B receive the intervention’s training and wrap-around support. We collected baseline data in October and November 2014 at the end of the school year. We conducted random assignment in January 2015. The intervention began in June 2015 and we began baseline analysis in June 2015. We will collect follow-up data at the end of 2015 and 2016.

This design allows us to answer three main sets of research questions. First, we will compare the outcomes of teachers and students in schools assigned to group A to those in schools assigned to group B. This first comparison allows us to provide unbiased estimates of the impact of the FA intervention, holding constant the EOG wrap-around support intervention. Second, we will compare the outcomes of teachers and students in schools assigned to group B to those in schools assigned to group C. This comparison allows us to provide unbiased estimates of the impact of the EOG wrap-around support intervention compared to the ongoing programming provided by MOE and other organizations. Third, we will also be able to compare the outcomes of teachers and students in schools assigned to group A to those in schools assigned to group C. This comparison allows us to provide unbiased estimates of the impact of the combined use of EOG wrap-around support and FA with support, compared to the ongoing programming provided by MOE and other organizations. We will estimate each intervention’s impact on process outcomes (teachers’ access to assessment materials and results), intermediate outcomes (teachers’ use of assessment results to modify instruction), and final outcomes (learning, as measured by EOG tests and an independent assessment). We will estimate impacts at the end of the first and second years of the interventions (the 2015 and 2016 academic years). We will evaluate impacts on teaching methods at the end of both years, and will evaluate impacts on reading and math test scores at the end of the second year of the intervention.

This study will also include an implementation study which will help demonstrate how the interventions are implemented in reality, and if the implementation differed from the original plans. The implementation study will consist of gathering and analyzing qualitative and quantitative data on what interventions were implemented, and how. Qualitative data collection will consist of key informant interviews and focus groups. In addition, if we find that the intervention is effective, we will also conduct cost-effectiveness analysis.

A rigorous evaluation of the impact of these types of assessments in Honduras will contribute to a growing body of evidence on what works to improve early grade reading in primary schools, and more generally will test the effectiveness and costs of promising reading and education-access interventions. This evaluation is part of a five-year project to rigorously evaluate and cost United States Agency for International Development (USAID) investments in early literacy and access to education in conflict settings in Latin America and the Caribbean (LAC). Mathematica is currently leading multi-year studies in four countries: Guatemala, Honduras, Nicaragua and Peru. This intervention and the evaluation of the intervention are funded by USAID.

Intervention

We undertake a randomized controlled trial (RCT) in 180 primary schools in Honduras, complemented by qualitative analysis, to estimate the impacts of providing print materials and pedagogical support to maximize teachers’ ability to take advantage of these assessments in early primary grades. To do this, we randomize schools into 3 experimental groups: EOG wrap-around support and FA with support (Group A), EOG wrap-around support only (Group B), and ongoing programming by Honduras’ Ministry of Education (MOE) (Group C, the control group). Group C has limited access to printed FA materials from the Ministry’s previous efforts to distribute them. All schools administer an EOG test, but only schools in Groups A and B receive the intervention’s training and wrap-around support. We collected baseline data in October and November 2014 at the end of the school year. We conducted random assignment in January 2015. The intervention began in June 2015 and we began baseline analysis in June 2015. We follow the cohort of students who were in the second grade in the first year of the 2-year intervention.

This design allows us to answer three main sets of research questions. First, we will compare the outcomes of teachers and students in schools assigned to group A to those in schools assigned to group B. This first comparison allows us to provide unbiased estimates of the impact of the FA intervention, holding constant the EOG wrap-around support intervention. Second, we will compare the outcomes of teachers and students in schools assigned to group B to those in schools assigned to group C. This comparison allows us to provide unbiased estimates of the impact of the EOG wrap-around support intervention compared to the ongoing programming provided by MOE and other organizations. Third, we will also be able to compare the outcomes of teachers and students in schools assigned to group A to those in schools assigned to group C. This comparison allows us to provide unbiased estimates of the impact of the combined use of EOG wrap-around support and FA with support, compared to the ongoing programming provided by MOE and other organizations. If group B schools receive FA support in practice, we will also compare outcomes for groups A and B schools combined to group C outcomes.

Experimental design

After identifying our sample of 180 schools, we randomly assigned each school to one of the three experimental groups described above (A, B or c) with equal probabilities. We implemented random assignment in two steps to ensure that the random assignment would result in three groups that were balanced geographically, in terms of baseline exposure to the interventions and in terms of baseline performance. First, we divided each of four departments' (states') schools into a high or low baseline use group, according to their score on an index measuring baseline use of formative and end of grade assessment data. This generated eight strata. Then, we sorted schools within each stratum by performance on the schools’ 2013 first grade reading end of grade test scores. We then created triplets of schools with similar test scores within each stratum. For example, the three schools with the best scores in the group of low exposure schools in Lempira would be one triplet, the next three schools would be another triplet, and so forth. In addition to improving balance across the three experimental groups on geographic areas, baseline exposure, and baseline performance, this approach is expected to improve the precision of our estimates.

Design details

Randomization method
Randomization was done in office by a computer using Stata's random number generation.
Randomization unit
The unit of randomization is the school.
Clusters
180 schools.
Observations
1800 students, 10 per school. Because some small schools will not have 10 students, we will sample some additional children in larger schools to achieve a sample size of 1800 students.
Arms
60 schools in experimental group A (EOG wrap-around support and FA with support), 60 schools in experimental group B (EOG wrap-around support only) and 60 schools in experimental group C (ongoing programming by Honduras’ Ministry of Education, control group).
Primary outcomes
We will estimate each intervention’s impact on: 1) process outcomes (teachers’ access to assessment materials and results) 2) intermediate outcomes (teachers’ use of assessment results to modify instruction) 3) final outcomes (academic achievement, as measured by end of grade tests and an independent assessment). We will study impacts on process and intermediate outcomes at the teacher and school levels, and will estimate final outcomes on learning at the student level. We will estimate impacts at the end of the first and second years of the interventions (the 2015 and 2016 academic years).

Cited by

No citation data available for this registration yet.

Keywords