Supporting Student Engagement with Remote Learning: A randomized controlled trial in Chicago Public Schools
How to cite
Momeni, Fatemeh; Bhatt, Monica; Guryan, Jonathan; Oreopoulos, Philip. 2020. "Supporting Student Engagement with Remote Learning: A randomized controlled trial in Chicago Public Schools." AEA RCT Registry. AEARCTR-0006250. https://doi.org/10.1257/rct.6250-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
With the disruption in classroom learning caused by the COVID-19 pandemic, many students may be falling behind in their learning trajectory and less prepared to begin their next grade in the fall. Some researchers are predicting a 30-50% loss in learning due to the COVID-19 pandemic across subjects and up to a year’s worth of learning loss in mathematics, and there is evidence to suggest that the pandemic will reify and exacerbate existing systemic inequalities. Most U.S. school districts have turned to remote learning to mitigate this learning loss, however, they face many challenges to ensuring that high quality content is delivered and utilized by students, including technical access, curriculum building, and student engagement. We are partnering with Chicago Public Schools (CPS) to conduct three large-scale randomized controlled trials to try to understand what works to engage and support students in a virtual summer learning program.
In the first RCT, we will randomize students eligible for the Summer Learning program who do not register by CPS’ deadline into intensive outreach treatment to learn how we can better encourage families to sign up for this program. In the second and third RCTs, we study how we can best increase student engagement with remote learning and the online Khan Academy platform offered by the Summer Learning program by randomizing students to receiving different forms of learning and engagement support. We hope the three studies will inform future planning and implementation of remote learning efforts in the district in the coming school year.
Intervention
BACKGROUND
With the disruption in classroom learning caused by the COVID-19 pandemic, many students may be falling behind in their learning trajectory and less prepared to begin their next grade in the fall. Some researchers are predicting a 30-50% loss in learning due to the COVID-19 pandemic across subjects and up to a year’s worth of learning loss in mathematics, and there is evidence to suggest that the pandemic will reify and exacerbate existing systemic inequalities. Since in many ways the only mode of delivery will be through remote learning, any proposals to mitigate this learning loss must involve consideration of access to hardware and connectivity. However, having sufficient technical access still does not solve perhaps the most important challenge, which is ensuring that high quality content is delivered and utilized by students.
The Chicago Public Schools (CPS) Summer Learning program is one proposal to address the remote learning needs of students and successfully prepare them for SY2021.
The CPS Summer Learning program offers students in grades 1-8 who struggled to engage with virtual learning in Spring 2020 with an important opportunity to re-engage with the material during the Summer and, ultimately, successfully transition to the next grade level in SY2021. CPS students are eligible for the Summer Learning program if they received an ‘Incomplete’ in Math, Reading, or both during the Spring 2020 academic semester. As part of the Summer Learning Program, CPS is utilizing the online Khan Academy platform, a free, computer-assisted learning (CAL) programs available to teachers, students, and parents to support students in mathematics. The program is designed to save teachers’ time, lower education costs, and offer a more customizable learning experience to students. The goal is to facilitate interactive learning while allowing students to progress at their own pace. Such a readily available, no-cost platform makes it a promising tool for CPS to scale district-wide.
However, for students to benefit from the Summer Learning program and its curriculum, they first must be equipped with the proper information, technology, and face-to-face support to not only register but successfully participate in this virtual program.
INTERVENTIONS
We focus on three primary questions for this program evaluation:
1. Do personal outreach efforts to families induce registration for Summer Learning?
2. Do text message reminders increase attendance in Summer Learning?
3. Do personalized supports such as attendance monitoring, outreach to families, and individualized tutoring sessions increase attendance and performance in Summer Learning?
In addition, we are interested in the following secondary research questions:
4. What is the synergistic effect, if any, of getting both the text message reminders and the personalized supports during the Summer Learning program?
5. For this population of students, do additional supports during a four-week summer learning program influence their engagement with online learning in the following school year?
We plan to answer these questions by conducting three large-scale Randomized Control Trials (RCTs).
In the first RCT (referred to as Experiment 1 hereafter), we will randomize students eligible for the Summer Learning program who do not register by CPS’ registration deadline to two treatment arms: outreach worker treatment and control group. This effort will provide CPS with the opportunity to better understand whether they can improve program take-up through personal outreach. For students randomly assigned to the outreach worker treatment, outreach workers will reach out to students’ families using existing CPS contact information during the two weeks prior to the Summer Learning program. During such outreach, outreach workers will inform families of the Summer Learning program of their child’s eligibility to participate in the program; answer any questions they may have about the program; collect information on student barriers to virtual learning and/or any hesitations for registering for the program; and assist them with filling out the registration form. The control group will consist of randomly assigned students who will not be reached out to by outreach workers after the registration deadline is passed.
In the second RCT (referred to as Experiment 2 hereafter), which takes place while Summer Learning program is in session, we will randomize students who have registered for the Summer Learning program and consented to receive text messages into two treatment arms: text message reminders and control group. The text message reminders will be sent to parents to remind them of Summer Learning class times and to encourage them to have their child join their Summer Learning sessions.
In the third RCT (referred to as Experiment 3 hereafter), which also takes place during the Summer Learning program, we will randomize Summer Learning classrooms into two treatment arms: resident teacher support and control group. Resident teachers, who are teachers-in-training employed by CPS, will be randomly assigned to Summer Learning classrooms and will provide support to lead classroom teachers to increase student engagement and attendance at daily Summer Learning sessions; act as a mentor/role model to students and develop positive relationships with students in their assigned class(es); and support students to improve the use and efficacy of their online math learning platform – Khan Academy.
Experiment 2 and Experiment 3 start after completion of Experiment 1 and set up a 2x2 randomization scheme where students may receive the Resident Teacher treatment only, the texting treatment only, the Resident Teacher and texting treatment, or no treatment (control).
Experimental design
Experiment 1: We will evaluate the impact of outreach workers on Summer Learning registration using a randomized controlled trial. Students who were eligible for CPS’ Summer Learning program were randomized into either the outreach worker treatment group or the control group. Each of the 20 outreach workers received a random list of 400 students and proceeded to call parents working down their lists over the course of one week.
Experiment 2: We will evaluate the impact of text message reminders on Summer Learning participation and academic outcomes. Students whose parents registered them for Summer Learning and consented to receive text messages from CPS were randomized into either the text message reminder treatment or the control group. 3,804 students were randomized into the treatment group and 3,804 were randomized into control.
Experiment 3: We will evaluate the impact of classroom support in the form of a Resident Teacher on student Summer Learning participation and academic outcomes. We randomized Resident Teacher support at the classroom level. Out of 461 Summer Learning classrooms, 120 were randomly assigned Resident Teacher support.
Experiment 2 and Experiment 3 set up a 2x2 randomization scheme where students may receive the Resident Teacher treatment only, the texting treatment only, the Resident Teacher and texting treatment, or no treatment (control).
Design details
Cited by
No citation data available for this registration yet.