The Effects of a Tech-Based Tutoring Program in Ukraine

How to cite

Dinarte, Lelys; Lemos, Renata; Gresham, James; Patrinos, Harry; Rodriguez, Rony. 2022. "The Effects of a Tech-Based Tutoring Program in Ukraine." AEA RCT Registry. AEARCTR-0010634. https://doi.org/10.1257/rct.10634-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Lelys Dinarte* , Renata Lemos , James Gresham , Harry Patrinos , Rony Rodriguez

* Primary investigator

Registry ID
AEARCTR-0010634
DOI
https://doi.org/10.1257/rct.10634-1.1
First registered
Last updated
Status
On going
Trial dates
2023-01-02 – 2024-08-31
Countries
Ukraine ()
Registry record
http://www.socialscienceregistry.org/trials/10634

Abstract

Evidence suggests that traumatic events, such as pandemics and wars, can impact children’s learning, socio-emotional development, and sense of protection (Quintana-Domeque and Ródenas-Serrano, 2017; Almond et al. 2018). Ukraine’s education system faces critical constraints in providing high-quality education to its students on their path toward recovery after disruptions to schooling and learning due to years of pandemic-related school closures. While children in many countries have gone back to school, the return to in-person education has been hindered by a lack of security, significant student and teacher displacement, and school damages posed by 6 months of the Russian invasion. Currently, learning losses in Ukraine are estimated to be over one year (Angrist et al, 2022), with learning outcomes falling below the lowest-performing countries in Europe which will have substantial impacts on human capital development in the country. To mitigate the impact of these traumatic events on children, the Ukrainian education system must find new strategies for supporting learning recovery and increasing learning equity while children are not able to return to in-person schooling.

High dosage small group instructional tutoring can be a powerful strategy to improve learning outcomes and cognitive and socioemotional skills, as it offers students a massive increase in personalized instruction, enabling teaching at the right level (Banerjee et al., 2015). To test the effectiveness of these programs in a conflict-affected setting, we study a tutoring program offering supplemental learning in math and Ukrainian language and psychosocial support. The target population of the tutoring program is Ukrainian students in grades 5 to 10 who are seeking supplemental support beyond the standard online schooling schedule. Initially, students are placed in groups of 3 and will receive 3 hours of tutoring per week for 6 weeks by paid-for tutors through an online platform. The implementing partner is Teach for Ukraine (https://teachforukraine.org/en/).

The impact evaluation will include three waves, through which we will test for the effectiveness of varying certain attributes, including program length, group size, content allocation, group composition by ability, and tutors as role models. Students will be recruited in each wave. All waves will have one control and one treatment group.

The effects of the tutoring program will be measured in math and Ukrainian language test scores, socioemotional skills, and mental health. As secondary outcomes, we will measure the effects of the intervention on expectations, time use, attendance to the tutoring activities, and attitudes towards tutoring.

Intervention

The intervention under evaluation consists of a tutoring program offering supplemental learning in math and Ukrainian language and psychosocial support. The learning component includes subject-specific academic content in accordance with Ukrainian educational programs. The psychosocial support component includes activities to enhance student’s social and emotional well-being. The target population of the tutoring program is Ukrainian students in grades 5 to 10 who seek supplemental support beyond the standard online schooling schedule.

The structure of the intervention will vary in each wave. In wave 1 of the intervention, we implement a "Barebones Program" where the group size is three students, the number of hours per week is 3, and the number of subjects is 2 (math and Ukrainian language).

In wave 2, we keep the same structure of the program as in wave 1, but we implement a variation--that we call the "enhanced program"-- with three additional features. First, we conduct a short assessment during the enrolment stage to have a proxy for student knowledge prior to the beginning of the tutoring program and used this measure to rank students and sort them in groups based on ability levels. Second, we use the information in this short assessment to prepare and share a diagnostic report with tutors for each group assigned to them, containing aggregated information on how well each group performed relative to the average for all students. Third, we developed short curriculum-based formative assessments and shared with tutors during the second week of the program to be used as a guide for tutors to assess and learn how well students were doing on several learning outcomes.

In the third wave, we use the same structure of the program as in wave 2 but we work with a team of psychologists and specialists from the Harvard Program in Refugee Trauma (HPRT) to enhance the psychosocial support component. In the updated curriculum for this component, we include activities related to breathing exercises, mental check ins, parables, and other activities.

Experimental design

The impact evaluation will be conducted in three waves of tutoring. In each wave, we will test for the effectiveness of varying certain attributes and following an agile experimenting approach. The exact attribute to be varied and tested in each wave will depend on the results from the previous wave.

Students will be recruited in each wave. The tutoring program will be advertised on social network platforms and disseminated by the Ministry to schools around the country, influencers in the Education Sector, and TFU. Students interested in taking part in the tutoring program will register online, provide responses to a questionnaire, and provide their own consent as well as parental consent. Students who submit this information will become part of the study sample and, for all the 3 waves, will be randomly assigned to the treatment or the control group.

The randomization procedure for each wave will consist of a two-stage stratified household-level randomization. In each wave, at the first stage, all enrolled students are randomly assigned to treatment or control groups. The stratification variables are whether the parent has completed higher education and if they were living in Ukraine for wave 1 and the interaction between region (Central, Eastern, Southern, Western, Out of Ukraine) and if the parent has completed higher education in waves 2 and 3. Those in the treatment groups receive the tutoring activities, and the ones in the control groups enrolled in the online platform and were allowed to interact among each other but did not receive the tutoring sessions.

For the second stage, we assign students to different tutoring groups stratifying by grade and preferred schedule in wave 1. The stratification variables in waves 2 and 3 included grade and preferred schedule and they were also ranked by ability using a short knowledge test in math and Ukrainian language collected during the registration period. Students are then sorted into small groups based on their ranking position.

Design details

Randomization method
Randomization will be done in the office using a code developed by the PIs
Randomization unit
The unit of randomization will be the household. Yet, based on pilot data, the average number of students enrolled per household is 1.2. Therefore, we expect our randomization to be almost at the individual level.
Clusters
At least 8,000 households
Observations
At least 9,600 students
Arms
Treatment --> 4,000 households; 4,800 students Control --> --> 4,000 households; 4,800 students
Primary outcomes
Math test score Ukrainian language test score Mental health
Secondary outcomes
Social emotional skills Personal and Academic Expectations Time use Attendance of the tutoring activities Attitudes towards tutoring
Sponsors
Name: The World Bank Url: https://www.worldbank.org/en/home Name: UBS Foundation Url: https://www.ubs.com/global/en/ubs-society/philanthropy/optimus-foundation.html
Partners
Name: Teach for Ukraine Partner_type: ngo Url: https://teachforukraine.org/en/

Cited by

No citation data available for this registration yet.

Keywords