Impact of Short-Term Tutoring on Cognitive and Non-Cognitive Skills in Chile
How to cite
Gallego, Francisco; Cuesta, José I.; Cabezas, Veronica. 2017. "Impact of Short-Term Tutoring on Cognitive and Non-Cognitive Skills in Chile." AEA RCT Registry. AEARCTR-0001480. https://doi.org/10.1257/rct.1480-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
A randomized evaluation in Chile finds that providing a 3-month program of small group tutoring to fourth graders using college student volunteers raise language scores and measures of attitudes towards reading for some subgroups of the population. In particular, students from low-performing and poor schools in areas in which the program was implemented increased their performance in a reading test by between 0.15 and 0.20 standard deviations and improved significantly their self-perceptions as readers. Researchers also present cost-effectiveness analyses and find that for the students for which the program was effective the program was in the range of the current estimates.
Intervention
In order to improve reading comprehension and attitudes toward reading in fourth grade students from vulnerable schools, the Chilean Ministry of Education partnered with the Fundación para la Superación de la Pobreza (FSP) in 2010 to implement Servicio Pas en Educación (SPE). In the poorest counties of the Santiago and Bío-Bío regions, the Chilean Ministry of Education selected 85 public and private schools that had scored below the average on the language portion of Chile' s national standardized test. From these 85 schools, 45 schools were randomly selected to receive SPE and the remaining 40 served as comparison schools.
SPE brought college student volunteers into primary schools, where they conducted 90-minute reading sessions with groups of 5 or 6 fourth grade students. The sessions included a mix of traditional stories and instructional texts appropriate for the students' age. Volunteers were managed by a paid employee of FSP stationed permanently in each program school to ensure accurate implementation and provide assistance on effective teaching techniques. The program lasted for 15 weeks.
Surveys gathered information on reading comprehension and attitudes towards reading, as well as program implementation details, and individual school and student characteristics. The implementation of the program was monitored through random visits to observe tutoring sessions and administrative data on student, volunteer, and supervisor attendance.
Experimental design
The Servicio País en Educación (SPE) program emerged from a partnership between the Chilean Ministry of Education (Mineduc) and the Fundación para la Superación de la Pobreza (Poverty Alleviation Foundation, FSP) and was first implemented during the period between September and December of 2010. The main objective of the program was to improve both attitudes toward reading (IR) and reading comprehension (RC) in 4th graders from vulnerable schools. Additionally, the program aimed at intervening in two additional dimensions: to generate new learning environments in which to implement the program, being this aspect influenced by the earthquake that affected the central region of Chile in February, 2010. Secondly, SPE intended to include college students as direct intervention agents. However, this evaluation only answers the questions related to the impact of the program on IR and RC.
The SPE program targeted vulnerable schools of the central region of Chile. Additionally, a number of logistic restrictions were put by FSP. Researchers excluded all schools from counties in which the FSP either was not able to work because they had no human resources in them or had already committed with some schools in it, which made randomization impossible there. This reduced the number of counties from which schools were included in the evaluation to 10: Santiago, Estación Central, Lo Espejo, Maipú, La Florida and San Bernardo from the Great Santiago region (Metropolitan Region, RM), and Concepción, Coronel, Hualpén and Talcahuano from the Biobio Region (VIII). In some of those counties, an additional restriction was set in terms of the administrative dependence of the schools, restricting us to include either only public schools (P) in some counties or only private subsidized schools (PS) in other ones. In addition, to fit the operational model designed by the FSP, each of the included schools had to have at least 90 students in fourth grade.
Using this sample, schools were randomly assigned to treatment and control groups, stratifying by county, socioeconomic group and SIMCE test scores. As the eligible school set was larger than the number of required schools, only some of the schools assigned to each group were included in the evaluation, decision that was random too. The remaining schools were kept as replacement lists for the eventual rejection of schools to take part of the evaluation. With the results of this assignment, schools were contacted and invited to take part of the evaluation in their corresponding group. Five of them rejected the program but, except in two cases, all of them accepted to be evaluated anyway. Additionally, two schools in the control group rejected to be evaluated. All of these schools were randomly replaced by another schools coming from the replacement lists. The treatment and control groups were finally composed of 45 and 40 schools respectively, grouped in 25 and 24 units.
Baseline data on the students were collected before the start of the program in August 2010, and follow-up data were collected after the program finished in December 2010. Researchers included two instruments. (1) The Prueba de Comprensión Lectora y Producción de Textos (Reading Comprehension and Texts Production Test, CLPT) measures formal reading skills, i.e., Reading Comprehension (RC), Texts Production (TP), and use of Language (UL). (2) The Gusto por la Lectura (Taste for Reading, GPL) measures attitudes towards reading along the four dimensions: Interest for Reading (IR), Self- perception as a Reader (SPR), Enjoyableness for Reading (ER) and Perception of Reading at School (PRS). These four indexes move discretely between 0 and 3, where 0 is the most negative of the offered alternatives and 3 is the most positive of the offered alternatives. Researchers also collected information on the program operation to understand the reasons behind the heterogeneous impacts of the program in different subpopulations.
First, researchers monitored the implementation of the program with random visits to observe the actual tutoring sessions. Second, they gathered administrative information about student, tutor, and professional assistance to tutoring sessions and about the numbers of sessions received by each student in the program. Finally, researchers collected data from the Ministry of Education on both schools’ and students’ and characteristics. Regarding schools, researchers collected data on Language and Math SIMCE test scores, average mothers’ years of schooling, average household income, socioeconomic level, a school vulnerability index (IVE), and administrative dependence. Regarding students, researchers collected information on gender, grades for 2009 and school attendance for 2009 and 2010, coming them from Mineduc records too.
Researchers estimated the effect of the program by comparing average outcomes of the treatment and the control group. Additionally, they perform some statistical exercise in order to understand the relationship between the two tests results. To assess the Intention-to-Treat (ITT) and the Treatment-on-the-Treated (TT) effects on the CLPT scores, researchers use OLS models. For TT, researchers use number of sessions of the program received by a student, which stands as a measure of the intensity of the program, and which researchers instrument using the intention-to-treat dummy as an instrumental variable (IV). To estimate the ITT and TT effects of the program on GPL test scores, researchers used ordered logit models. To estimate the TT effect, researchers used the number of sessions of the program received by the student to control for endogeneity. Researchers also present cost effectiveness analyses.
Design details
Papers from this trial
- Cabezas, Veronica, Jose Cuesta, and Francisco Galleg. "Effects of Short-Term Tutoring on Cognitive and Non-Cognitive Skills: Evidence from a Randomized Evaluation in Chile." Working Paper, May 2011. https://www.povertyactionlab.org/sites/default/files/publications/493%20-%20short-term%20tutoring%20May2011.pdf
Cited by
No citation data available for this registration yet.