Sustainability Impact Evaluation of the Early Grade Reading Study (EGRS I) and Impact Evaluation of the Reading Support Project (RSP)

How to cite

Roper, Margaret; Williams, Benita; Jukes, Matthew; Jonathan, Stern; Mohohlwane, Nompumelelo; Taylor, Stephen. 2021. "Sustainability Impact Evaluation of the Early Grade Reading Study (EGRS I) and Impact Evaluation of the Reading Support Project (RSP)." AEA RCT Registry. AEARCTR-0008242. https://doi.org/10.1257/rct.8242-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Margaret Roper* , Benita Williams , Matthew Jukes , Stern Jonathan , Nompumelelo Mohohlwane , Stephen Taylor

* Primary investigator

Registry ID
AEARCTR-0008242
DOI
https://doi.org/10.1257/rct.8242-1.1
First registered
Last updated
Status
Completed
Trial dates
2015-05-01 – 2022-06-30
Countries
South Africa (North West Province, Districts of Ngaka Modiri Molema and Dr. Kenneth Kaunda)
JEL codes
Analysis of Education I21
Registry record
http://www.socialscienceregistry.org/trials/8242

Abstract

The South African Department of Basic Education has tested interventions such as scripted lesson plans, teacher training and coaching to improve reading proficiency in learners taught in African home languages by teachers in under resourced schools. The EGRS I was an RCT implemented between 2015 and 2017 which tested three interventions: Treatment 1: Training, scripted lessons, graded readers, Treatment 2: Reading Coaches, scripted lessons, graded readers, Treatment 3: Parental involvement Programme. The EGRS I impact evaluation showed that the coaching treatment resulted in statistically significant improvements in African Home Language reading proficiency. A 2019 sustainability assessment of the EGRS I, indicated that reading proficiency gains were sustained after learners were no longer taught by supported teachers. In response to these encouraging findings, the Reading Support Programme was implemented to test whether a similar intervention can be implemented on a larger scale. The RSP involved all Foundation Phase teachers for two years (The EGRS I targeted one grade’s teachers per annum and provided one year’s support), and delivered the intervention in both Setswana Home Language as well as English First Additional Language (The EGRS I only provided support in Setswana Home Language). Similar to the EGRS, the RSP was designed to test if there were any reading proficiency gains in schools that benefitted from teacher training and scripted lessons, and there was a differential gain in those schools that also benefitted from literacy coaching support and those that benefitted from a leadership development intervention targeted at School Management Team members.

This 2021 impact evaluation will investigate 1) The impact of EGRS interventions on a cohort of learners tracked for six years from the beginning of Grade 1 (on interventions that ended in either Grade 2 or Grade 3). 2) The impact of RSP interventions (and Classroom Library Program) on a cohort of learners tracked for two years from Grade 1. 3) The impact of RSP interventions (and Classroom Library Program) on learners in Grade 4 in 2021. 4) The impact of both EGRS and RSP on a cohort of learners tracked for two years from Grade 1.

This is a continuation of the Early Grade Reading Study in South Africa, 2016, RCT ID AEARCTR-0001749.

Intervention

DESCRIPTION OF THE EGRS I
This EGRS I was set up as a randomized control trial. The EGRS included three different interventions, all aimed at improving early-grade reading in Setswana. These three interventions were implemented with the teachers of a cohort of learners in Grade 1 in 2015, the teachers of the same cohort of learners in Grade 2 in 2016, and the first two interventions were extended to the teachers of the same learners again in Grade 3 in 2017.
Treatment 1: Training, scripted lessons, graded readers
Treatments 1 and 2 aimed to apply the same set of instructional practices in the teaching of home language literacy in grade 1, 2 and 3 classrooms working in one grade at a time. Both treatments therefore provided teachers with clearly scripted lesson plans, which were aligned to the curriculum as specified in the Curriculum and Assessment Policy Statements (CAPS) for home language literacy in the Foundation Phase. The lesson plans incorporated the use of learning support materials including the government-provided workbooks as well as certain additional materials (graded reading booklets, flash cards, posters, etc.), which were provided through the EGRS. The graded reading booklets provided a key resource for the teacher to use in group-guided reading and individual work so as to facilitate reading practice at an appropriate pace and sequence of progression.
Treatment 1 trained the teachers on how to use the lesson plans and accompanying materials through central training sessions, each lasting two days, and occurring twice yearly.
Treatment 2: Reading Coaches, scripted lessons, graded readers
Exactly the same set of instructional materials (scripted lesson plans, graded reading booklets and other materials) were provided to Treatment 2 schools. However, instead of central training sessions, one day training/orientation was provided at the start of each term, accompanied by ongoing support to teachers consisting of regular (monthly) on-school coaching from specialist “reading coaches” visits. In addition to these on-site visits, there were occasional needs-based workshops with the coach and a small cluster of nearby Treatment 2 schools.
Treatment 3: Parental involvement
Treatment 3 was designed to promote parental involvement to support their children’s reading progress. At each of the 50 schools in this treatment arm a Community Reading Coach (CRC) was recruited. The CRC was identified through communication with the school principal who recommended a suitably qualified, but available person in the community. The CRCs attended a one-day training session facilitated by the service provider (Class Act) at the start of each school term (quarterly). The CRCs were trained to deliver weekly training sessions for Grade 1 parents at their respective schools. A total of 30 sessions were scheduled for each year covering a total of 10 topics. The parental involvement intervention arm was prematurely ended in 2016, after two years of implementation working first with Grade 1 parents and then Grade 2 parents. The parents of Grade 3 learners were not part of the intervention.
DESCRIPTION OF THE RSP
The RSP was set up as a randomized control trial, with six focus areas.
1. Quarterly ‘just in time’ training for teachers (from 263 schools) on the implementations of Curriculum and Assessment Policy Statement (CAPS) and lesson plans (all 263 schools)
2. The provision of Learning and Teaching Support Material (LTSM) packages (all 263 schools).
3. The provision of 14 literacy coaches to 140 schools to offer classroom-based support to Foundation Phase teachers (Coaching in 140 schools)
4. The development of the leadership capacity of principals/deputies and Head of Departments (HODs) to promote a culture of reading in their schools (SMT training in 65 schools)
5. Through the DBE, Classroom libraries are provided to 98 schools
The RSP specifically aims to: improve subject matter knowledge; promote more effective pedagogic practices; improve in-class time management; increase effective use of LTSM; and foster a school environment to support teachers’ ability to implement the full curriculum and facilitate successful teaching and learning.
INTERACTIONS BETWEEN EGRS I AND RSP
In order to set up research questions and analyses accurately, it is helpful to understand how the EGRS and RSP interventions may interact and how they compare, in three areas.
1) Teacher Training
Teacher training for EGRS I focused on one language (Setswana) and trained one grade of teachers each year, for one year (Grade 1, then Grade 2, then Grade 3). RSP trained all three groups of teachers at the same time for two years in both Setswana and English. We may expect interactions between the two interventions in that some teachers may have received training in Setswana instructions under both programs.
2) Coaching
Coaches used in the EGRS program were not reemployed under RSP. Thus, we expect interactions to be limited in the coaching intervention of the two programs. It is possible that teachers who received coaching under EGRS may have been more experienced in getting feedback and may have benefited more from the RSP coaching as a result.
3) Learner and Teacher Support Materials
These materials were provided under both programs. In EGRS they were only provided for Setswana while the availability of EFAL materials to teachers was not provided and it is not clear what materials were used. In RSP materials were provided for both Setswana and EFAL. These included structured lesson plans, readers and other supporting LTSM. It is possible that the EGRS teachers were better prepared for LTSM use and further leveraged this when receiving RSP materials for both languages.
 
3.2. PRIMARY OUTCOMES
The primary outcome of interest is reading proficiency in Setswana home language (HL) and English first additional language (EFAL). For reading proficiency we will measure all the intermediate steps towards comprehension: letter recognition, phonetic awareness, word recognition and reading, vocabulary, reading fluency, and reading with comprehension. These tests are adapted from standard tests that have already been developed for the English First Additional language and Setswana language, such as the Early-Grade-Reading-Assessment (EGRA), but also newly developed and extensively piloted reading tasks. As our main outcome indicator, we will construct an aggregate indicator of learning proficiency, using principal component analysis.

Experimental design

EVALUATION DESIGN EGRS I
The EGRS I was designed as a randomized control trial (RCT) with three treatment arms and one control arm. A total of 230 schools were randomly assigned to one of the intervention groups (50 schools each), or the control group (80 schools). To evaluate the effectiveness of the interventions, a random sample of 20 learners were selected in Grade 1 in 2015 and were tracked into Grade 2. Learners’ reading proficiency was tested at the start of Grade 1, at the end of Grade 1, and again at the end of Grade 2.
Treatment Assignment and Random Selection for EGRS I impact evaluation
The sampling process for the EGRS I began with 458 primary schools registered in 2014 administrative data in the districts of Dr. Kenneth Kaunda and Ngaka Modiri Molema. The sampling started by excluding relatively affluent schools (those in quintiles 4 and 5). Next, schools were excluded in which the language of instruction in the Foundation Phase was not Setswana. Schools were also excluded if they were missing in the 2014 Annual National Assessment (ANA) dataset. An additional eight schools were excluded because they took part in the EGRS pilot. Small schools (fewer than 20 Grade 1 enrolments) were also excluded because many of them practice multi-grade teaching rendering the scripted lesson plans less appropriate. Larger schools (more than 180 Grade 1 enrolments) were also excluded to limit intervention costs. Three more schools were excluded after the North West PED checked the list of schools and found specific problems with these schools (e.g., the school had been closed down, or a particular conflict around school management was occurring in a school). After all of these exclusions 235 eligible schools remained. Using a random number generator, five schools were selected as possible replacement schools, with the remaining 230 schools constituting the main sample.
To increase power and assure balance between treatment arms, stratified randomization was performed. Ten strata of 23 similar schools each were created, based on school size, socio-economic status, and previous performance in the Annual National Assessments. Within each stratum, five schools were randomly assigned to each treatment group and eight to the control group. In total, 50 schools were assigned to each treatment group and 80 to the control group. Sample size calculations indicated that this sample is sufficient to identify a minimum effect size of 0.21 standard deviations when comparing a treatment group with the control group and a minimum effect size of 0.23 standard deviations when comparing two treatment groups. These calculations are based on data collection from 20 Grade 1 learners and assume a 95% confidence interval, an alpha value of 0.8, an intra-class correlation coefficient (rho) of 0.3 and a correlation between pre- and post-test scores of 0.7.
EVALUATION DESIGN RSP
A sample of 214 schools that were part of the EGRS and part of the RSP 2018 baseline data collection is included in the impact evaluation. This includes all intervention groups. Lastly, 15 schools that were part of the original EGRS sample were not included in the RSP baseline because they had converted to multi-grade schools.
Schools were assigned to RSP treatment groups as follows. The 50 schools that were in the parent group under EGRS were assigned to be the comparison group for RSP, because the parent intervention had no impact. The comparison group received training and LTSM only. In addition, 80 schools that were not part of EGRS received the same (comparison) interventions under RSP. This group is not part of the RSP evaluation design, as they were not included in the baseline evaluation and have not received an intervention to be evaluated. Schools in the three remaining EGRS groups - Training (50 schools), Coaching (50 schools) and Control (80 schools) - received training and coaching under RSP. In addition, half of the schools in each group were allocated randomly to receive a School Management Training (SMT), in addition to coaching. Without schools dropping out, this procedure should have resulted in 90 schools receiving coaching alone, 90 schools receiving coaching and SMT and 50 schools receiving only training and LTSM, received by all other groups. However, approximately 60 schools dropped out of the study, according to the most recent (January 2021) DBE database. Table 2 shows the number of schools remaining in the RSP and EGRS groups.
See Tables 2 and 3 in the analysis plan for a provisional breakdown of the Number of schools in EGRS and RSP treatment groups

Design details

Randomization method
Schools were randomly assigned using a random number generator in statistical software (STATA).
Randomization unit
Randomization was conducted at the level of school, and at the level of the learner
Clusters
EGRS Grade 6 learners in 229 schools RSP Grade 3 learners in 214 schools RSP Grade 2 learners in 214 schools
Observations
12 EGRS Grade 6 learners in 229 schools = 3664 observations 16 RSP Grade 3 learners in 214 schools = 3424 observations 16 RSP Grade 2 learners in 214 schools = 3424 observations Parents of learners assessed in 229 schools = 10283 2 Grade 3 teachers in each of the 229 schools = 458 observations 2 Grade 7 teachers in each of the 229 schools = 458 observations 1 Principal in each of the schools = 229 observations
Arms
EGRS Treatment 1 schools (Training, scripted lessons, graded readers) = 80 EGRS Treatment 2 schools (Reading coaches, scripted lessons, graded readers) = 50 EGRS Treatment 3 schools (parent involvement) = 50 EGRS Control schools = 50 RSP Treatment 1 schools (Training, learning and teaching materials) = 43 RSP Treatment 2 schools (Training, learning and teaching materials, coaching) = 140 RSP Treatment 3 schools (Training, learning and teaching materials, coaching, school management training i.e., a subset of Treatment 2 schools) = 70
Primary outcomes
The primary outcome of interest is reading proficiency in Setswana home language (HL) and English first additional language (EFAL). For reading proficiency we will measure all the intermediate steps towards comprehension: letter recognition, phonetic awareness, word recognition and reading, vocabulary, reading fluency, and reading with comprehension. These tests are adapted from standard tests that have already been developed for the English First Additional language and Setswana language, such as the Early-Grade-Reading-Assessment (EGRA), but also newly developed and extensively piloted reading tasks. As our main outcome indicator, we will construct an aggregate indicator of learning proficiency, using principal component analysis.
Secondary outcomes
For the evaluation of RSP we will assess the role of intermediate outcomes in achieving impact in two categories – fidelity of implementation and mediators. Fidelity of Implementation The fidelity of implementation analysis will aim to understand whether schools implementing the interventions to a higher standard achieved greater improvements in reading outcomes. To do this, we will identify candidate variables from the contextual tools during the data collection pilot, in addition to data already collected from the RSP implementation to indicate fidelity of implementation (e.g., teacher use of national workbook) and assess the relationship between each of these variables and reading outcomes. It is not possible to identify a causal relationship between fidelity of implementation and reading outcomes because of confounding factors. To control for observed confounding factors, we will include baseline characteristics of schools as covariates in these analyses. Data available from the RSP implementation evaluation include: number of teachers per school attending training and receiving coaching and whether schools received project materials. Mediators The mediator analysis aims to identify the mechanism for the impact of interventions on outcomes. Mediators are typically school level processes (e.g., teacher corrects learner work) that are enhanced by the intervention and which lead to improved reading outcomes. Mediator variables that meet some or all of the following criterial will be considered. (1) The intervention has a significant impact on the mediator (2) There is a significant relationship between the mediator and reading outcomes (3) The impact of the treatment variables on HL and EFAL reading proficiency is attenuated when the mediator is included in the regression equation. Below are potential mediating variables (these will be determined during the final data analysis). Teachers Did the EGRS I and RSP treatment arms which involves training and coaching change teaching practice and effort? 1. Provide regular individualized assessment [TRQ – 3.7 and 3.10, CLW Section 2 document review)] 2. Require student to read out loud [TRQ – 3.7 and 3.10)] 3. Stream by ability [TRQ – 3.5] 4. Assess reading ability [TRQ – 3.7 and 3.10] 5. The completion of more writing exercises [TRQ – 3.7 and 3.10, CLW Section 2 document review] 6. The assessment of more writing exercises [CLW Section 2 document review] 7. Use of the national workbook [CLW Section 2 document review] 8. Increase the availability of home language text in classrooms [CLW Section 3] Parents Did EGRS I treatment arm lead to a change in parent attitudes and behavior? 1. Happy with child’s reading progress [PHQ -- 27] 2. Read to child [PHQ – 19] 3. Support child doing homework [PHQ – 30 and 31] Pupils 1. Pupil attendance [PRQ – 25] 2. Pupil do homework on days not at school [PHQ – 29] Heterogeneous Impacts At the pupil level, we could expect two opposing heterogeneous treatment effects of the EGRS and RSP interventions, based on baseline pupil reading proficiency. The scripted lesson plans require streaming by ability within the same classroom and provides opportunity for individualized attention and could benefit children who have otherwise been left behind. However, at the same time the scripted lesson plans are aligned to the national curriculum, which prescribes an ambitious pace in the South African context. The worst-performing pupils might benefit less if the teachers who follow the scripted lesson plans now progress at too fast a pace. Furthermore, boys/girls might benefit more/less from the individualized attention. Finally, the emphasis on individualized attention and tracking means that pupils might benefit more from the scripted lesson plans when the class size is large. For the RSP evaluation, we expect that the success of the interventions depend on teacher motivation, prior levels of effort, and ability. The scripted lesson plans will only be applied by teachers who have a sufficient level of intrinsic motivation. Related, teachers who have a higher burden of lesson preparation are most likely to switch to scripted lesson plans, because they have most to gain from scripted lesson. Furthermore, teachers may need a sufficient baseline level of reading proficiency in order to effectively apply the scripted lesson plans. On the other hand, the scripted lesson plans might be too restrictive for exceptional teachers who are effective at adjusting their instruction to the needs of the classroom and ability of the pupils. We will therefore examine the following possible interactions: Learner level 1. Learner gender [learner information form] 2. Learner age [learner information form] 3. Learner attendance of grade R [PHQ 15] 4. Learner attendance of other ECD [PHQ 18] 5. Learner baseline performance [pupil baseline test] 6. Learner attendance [PHQ 25, LWQ 2] 7. Learner speaks English at home [LWQ B1 8. Learning when not at school [PHQ 29 ,30; LWQ 2.2] 9. Learner reported amenities and possessions [LAL 1] 10. Learner reported books in home [LAL 2] Teacher level 1. Teacher characteristics: a. Education level [TRQ 2.5] b. Training in Foundation Phase [TRQ 2.8] c. Years of experience [TRQ 2.3] 2. Teacher Attendance [TRQ 4.1] 3. The extent to which teachers feel supported and recognized in their work [TRQ 5.2l] 4. The sufficiency of the classroom LTSM [TRQ 3.1, 3.2, 3.3., 3.4] School-level 1. Socio-economic background of school [administrative data] 2. School resources [SFO Section 2] 3. School urban / rural [administrative data] 4. School district [administrative data] 5. Pupil-teacher ratio [administrative data, SPQ section 3] 6. School days lost [SFO 5.1, SPQ 7.5, 7.7, 7.8, 7.11)] 7. Teacher absenteeism [SPQ 5.1] Parent level 1. Identity of the parent/guardian: [PHQ 8] E.g., single mother/grandmother; child-headed household. 2. Education level [PHQ 11,12] 3. Parent employment [PHQ 13] 4. Household amenities and possessions [PHQ14] 5. Learner use of English at home [PHQ 21, 22]

Cited by

No citation data available for this registration yet.

Keywords