Sustainability Impact Evaluation of the Early Grade Reading Study (EGRS I) and Impact Evaluation of the Reading Support Project (RSP)
How to cite
Roper, Margaret; Williams, Benita; Jukes, Matthew; Jonathan, Stern; Mohohlwane, Nompumelelo; Taylor, Stephen. 2021. "Sustainability Impact Evaluation of the Early Grade Reading Study (EGRS I) and Impact Evaluation of the Reading Support Project (RSP)." AEA RCT Registry. AEARCTR-0008242. https://doi.org/10.1257/rct.8242-1.1
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
The South African Department of Basic Education has tested interventions such as scripted lesson plans, teacher training and coaching to improve reading proficiency in learners taught in African home languages by teachers in under resourced schools. The EGRS I was an RCT implemented between 2015 and 2017 which tested three interventions: Treatment 1: Training, scripted lessons, graded readers, Treatment 2: Reading Coaches, scripted lessons, graded readers, Treatment 3: Parental involvement Programme. The EGRS I impact evaluation showed that the coaching treatment resulted in statistically significant improvements in African Home Language reading proficiency. A 2019 sustainability assessment of the EGRS I, indicated that reading proficiency gains were sustained after learners were no longer taught by supported teachers. In response to these encouraging findings, the Reading Support Programme was implemented to test whether a similar intervention can be implemented on a larger scale. The RSP involved all Foundation Phase teachers for two years (The EGRS I targeted one grade’s teachers per annum and provided one year’s support), and delivered the intervention in both Setswana Home Language as well as English First Additional Language (The EGRS I only provided support in Setswana Home Language). Similar to the EGRS, the RSP was designed to test if there were any reading proficiency gains in schools that benefitted from teacher training and scripted lessons, and there was a differential gain in those schools that also benefitted from literacy coaching support and those that benefitted from a leadership development intervention targeted at School Management Team members.
This 2021 impact evaluation will investigate 1) The impact of EGRS interventions on a cohort of learners tracked for six years from the beginning of Grade 1 (on interventions that ended in either Grade 2 or Grade 3). 2) The impact of RSP interventions (and Classroom Library Program) on a cohort of learners tracked for two years from Grade 1. 3) The impact of RSP interventions (and Classroom Library Program) on learners in Grade 4 in 2021. 4) The impact of both EGRS and RSP on a cohort of learners tracked for two years from Grade 1.
This is a continuation of the Early Grade Reading Study in South Africa, 2016, RCT ID AEARCTR-0001749.
Intervention
DESCRIPTION OF THE EGRS I
This EGRS I was set up as a randomized control trial. The EGRS included three different interventions, all aimed at improving early-grade reading in Setswana. These three interventions were implemented with the teachers of a cohort of learners in Grade 1 in 2015, the teachers of the same cohort of learners in Grade 2 in 2016, and the first two interventions were extended to the teachers of the same learners again in Grade 3 in 2017.
Treatment 1: Training, scripted lessons, graded readers
Treatments 1 and 2 aimed to apply the same set of instructional practices in the teaching of home language literacy in grade 1, 2 and 3 classrooms working in one grade at a time. Both treatments therefore provided teachers with clearly scripted lesson plans, which were aligned to the curriculum as specified in the Curriculum and Assessment Policy Statements (CAPS) for home language literacy in the Foundation Phase. The lesson plans incorporated the use of learning support materials including the government-provided workbooks as well as certain additional materials (graded reading booklets, flash cards, posters, etc.), which were provided through the EGRS. The graded reading booklets provided a key resource for the teacher to use in group-guided reading and individual work so as to facilitate reading practice at an appropriate pace and sequence of progression.
Treatment 1 trained the teachers on how to use the lesson plans and accompanying materials through central training sessions, each lasting two days, and occurring twice yearly.
Treatment 2: Reading Coaches, scripted lessons, graded readers
Exactly the same set of instructional materials (scripted lesson plans, graded reading booklets and other materials) were provided to Treatment 2 schools. However, instead of central training sessions, one day training/orientation was provided at the start of each term, accompanied by ongoing support to teachers consisting of regular (monthly) on-school coaching from specialist “reading coaches” visits. In addition to these on-site visits, there were occasional needs-based workshops with the coach and a small cluster of nearby Treatment 2 schools.
Treatment 3: Parental involvement
Treatment 3 was designed to promote parental involvement to support their children’s reading progress. At each of the 50 schools in this treatment arm a Community Reading Coach (CRC) was recruited. The CRC was identified through communication with the school principal who recommended a suitably qualified, but available person in the community. The CRCs attended a one-day training session facilitated by the service provider (Class Act) at the start of each school term (quarterly). The CRCs were trained to deliver weekly training sessions for Grade 1 parents at their respective schools. A total of 30 sessions were scheduled for each year covering a total of 10 topics. The parental involvement intervention arm was prematurely ended in 2016, after two years of implementation working first with Grade 1 parents and then Grade 2 parents. The parents of Grade 3 learners were not part of the intervention.
DESCRIPTION OF THE RSP
The RSP was set up as a randomized control trial, with six focus areas.
1. Quarterly ‘just in time’ training for teachers (from 263 schools) on the implementations of Curriculum and Assessment Policy Statement (CAPS) and lesson plans (all 263 schools)
2. The provision of Learning and Teaching Support Material (LTSM) packages (all 263 schools).
3. The provision of 14 literacy coaches to 140 schools to offer classroom-based support to Foundation Phase teachers (Coaching in 140 schools)
4. The development of the leadership capacity of principals/deputies and Head of Departments (HODs) to promote a culture of reading in their schools (SMT training in 65 schools)
5. Through the DBE, Classroom libraries are provided to 98 schools
The RSP specifically aims to: improve subject matter knowledge; promote more effective pedagogic practices; improve in-class time management; increase effective use of LTSM; and foster a school environment to support teachers’ ability to implement the full curriculum and facilitate successful teaching and learning.
INTERACTIONS BETWEEN EGRS I AND RSP
In order to set up research questions and analyses accurately, it is helpful to understand how the EGRS and RSP interventions may interact and how they compare, in three areas.
1) Teacher Training
Teacher training for EGRS I focused on one language (Setswana) and trained one grade of teachers each year, for one year (Grade 1, then Grade 2, then Grade 3). RSP trained all three groups of teachers at the same time for two years in both Setswana and English. We may expect interactions between the two interventions in that some teachers may have received training in Setswana instructions under both programs.
2) Coaching
Coaches used in the EGRS program were not reemployed under RSP. Thus, we expect interactions to be limited in the coaching intervention of the two programs. It is possible that teachers who received coaching under EGRS may have been more experienced in getting feedback and may have benefited more from the RSP coaching as a result.
3) Learner and Teacher Support Materials
These materials were provided under both programs. In EGRS they were only provided for Setswana while the availability of EFAL materials to teachers was not provided and it is not clear what materials were used. In RSP materials were provided for both Setswana and EFAL. These included structured lesson plans, readers and other supporting LTSM. It is possible that the EGRS teachers were better prepared for LTSM use and further leveraged this when receiving RSP materials for both languages.
3.2. PRIMARY OUTCOMES
The primary outcome of interest is reading proficiency in Setswana home language (HL) and English first additional language (EFAL). For reading proficiency we will measure all the intermediate steps towards comprehension: letter recognition, phonetic awareness, word recognition and reading, vocabulary, reading fluency, and reading with comprehension. These tests are adapted from standard tests that have already been developed for the English First Additional language and Setswana language, such as the Early-Grade-Reading-Assessment (EGRA), but also newly developed and extensively piloted reading tasks. As our main outcome indicator, we will construct an aggregate indicator of learning proficiency, using principal component analysis.
Experimental design
EVALUATION DESIGN EGRS I
The EGRS I was designed as a randomized control trial (RCT) with three treatment arms and one control arm. A total of 230 schools were randomly assigned to one of the intervention groups (50 schools each), or the control group (80 schools). To evaluate the effectiveness of the interventions, a random sample of 20 learners were selected in Grade 1 in 2015 and were tracked into Grade 2. Learners’ reading proficiency was tested at the start of Grade 1, at the end of Grade 1, and again at the end of Grade 2.
Treatment Assignment and Random Selection for EGRS I impact evaluation
The sampling process for the EGRS I began with 458 primary schools registered in 2014 administrative data in the districts of Dr. Kenneth Kaunda and Ngaka Modiri Molema. The sampling started by excluding relatively affluent schools (those in quintiles 4 and 5). Next, schools were excluded in which the language of instruction in the Foundation Phase was not Setswana. Schools were also excluded if they were missing in the 2014 Annual National Assessment (ANA) dataset. An additional eight schools were excluded because they took part in the EGRS pilot. Small schools (fewer than 20 Grade 1 enrolments) were also excluded because many of them practice multi-grade teaching rendering the scripted lesson plans less appropriate. Larger schools (more than 180 Grade 1 enrolments) were also excluded to limit intervention costs. Three more schools were excluded after the North West PED checked the list of schools and found specific problems with these schools (e.g., the school had been closed down, or a particular conflict around school management was occurring in a school). After all of these exclusions 235 eligible schools remained. Using a random number generator, five schools were selected as possible replacement schools, with the remaining 230 schools constituting the main sample.
To increase power and assure balance between treatment arms, stratified randomization was performed. Ten strata of 23 similar schools each were created, based on school size, socio-economic status, and previous performance in the Annual National Assessments. Within each stratum, five schools were randomly assigned to each treatment group and eight to the control group. In total, 50 schools were assigned to each treatment group and 80 to the control group. Sample size calculations indicated that this sample is sufficient to identify a minimum effect size of 0.21 standard deviations when comparing a treatment group with the control group and a minimum effect size of 0.23 standard deviations when comparing two treatment groups. These calculations are based on data collection from 20 Grade 1 learners and assume a 95% confidence interval, an alpha value of 0.8, an intra-class correlation coefficient (rho) of 0.3 and a correlation between pre- and post-test scores of 0.7.
EVALUATION DESIGN RSP
A sample of 214 schools that were part of the EGRS and part of the RSP 2018 baseline data collection is included in the impact evaluation. This includes all intervention groups. Lastly, 15 schools that were part of the original EGRS sample were not included in the RSP baseline because they had converted to multi-grade schools.
Schools were assigned to RSP treatment groups as follows. The 50 schools that were in the parent group under EGRS were assigned to be the comparison group for RSP, because the parent intervention had no impact. The comparison group received training and LTSM only. In addition, 80 schools that were not part of EGRS received the same (comparison) interventions under RSP. This group is not part of the RSP evaluation design, as they were not included in the baseline evaluation and have not received an intervention to be evaluated. Schools in the three remaining EGRS groups - Training (50 schools), Coaching (50 schools) and Control (80 schools) - received training and coaching under RSP. In addition, half of the schools in each group were allocated randomly to receive a School Management Training (SMT), in addition to coaching. Without schools dropping out, this procedure should have resulted in 90 schools receiving coaching alone, 90 schools receiving coaching and SMT and 50 schools receiving only training and LTSM, received by all other groups. However, approximately 60 schools dropped out of the study, according to the most recent (January 2021) DBE database. Table 2 shows the number of schools remaining in the RSP and EGRS groups.
See Tables 2 and 3 in the analysis plan for a provisional breakdown of the Number of schools in EGRS and RSP treatment groups
Design details
Cited by
No citation data available for this registration yet.