Supporting Student Engagement with Remote Learning: A randomized controlled trial in Chicago Public Schools

How to cite

Momeni, Fatemeh; Bhatt, Monica; Guryan, Jonathan; Oreopoulos, Philip. 2020. "Supporting Student Engagement with Remote Learning: A randomized controlled trial in Chicago Public Schools." AEA RCT Registry. AEARCTR-0006250. https://doi.org/10.1257/rct.6250-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Fatemeh Momeni* , Monica Bhatt , Jonathan Guryan , Philip Oreopoulos

* Primary investigator

Registry ID
AEARCTR-0006250
DOI
https://doi.org/10.1257/rct.6250-1.0
First registered
Last updated
Status
On going
Trial dates
2020-07-06 – 2021-06-01
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/6250

Abstract

With the disruption in classroom learning caused by the COVID-19 pandemic, many students may be falling behind in their learning trajectory and less prepared to begin their next grade in the fall. Some researchers are predicting a 30-50% loss in learning due to the COVID-19 pandemic across subjects and up to a year’s worth of learning loss in mathematics, and there is evidence to suggest that the pandemic will reify and exacerbate existing systemic inequalities. Most U.S. school districts have turned to remote learning to mitigate this learning loss, however, they face many challenges to ensuring that high quality content is delivered and utilized by students, including technical access, curriculum building, and student engagement. We are partnering with Chicago Public Schools (CPS) to conduct three large-scale randomized controlled trials to try to understand what works to engage and support students in a virtual summer learning program.

In the first RCT, we will randomize students eligible for the Summer Learning program who do not register by CPS’ deadline into intensive outreach treatment to learn how we can better encourage families to sign up for this program. In the second and third RCTs, we study how we can best increase student engagement with remote learning and the online Khan Academy platform offered by the Summer Learning program by randomizing students to receiving different forms of learning and engagement support. We hope the three studies will inform future planning and implementation of remote learning efforts in the district in the coming school year.

Intervention

BACKGROUND

With the disruption in classroom learning caused by the COVID-19 pandemic, many students may be falling behind in their learning trajectory and less prepared to begin their next grade in the fall. Some researchers are predicting a 30-50% loss in learning due to the COVID-19 pandemic across subjects and up to a year’s worth of learning loss in mathematics, and there is evidence to suggest that the pandemic will reify and exacerbate existing systemic inequalities. Since in many ways the only mode of delivery will be through remote learning, any proposals to mitigate this learning loss must involve consideration of access to hardware and connectivity. However, having sufficient technical access still does not solve perhaps the most important challenge, which is ensuring that high quality content is delivered and utilized by students.

The Chicago Public Schools (CPS) Summer Learning program is one proposal to address the remote learning needs of students and successfully prepare them for SY2021.

The CPS Summer Learning program offers students in grades 1-8 who struggled to engage with virtual learning in Spring 2020 with an important opportunity to re-engage with the material during the Summer and, ultimately, successfully transition to the next grade level in SY2021. CPS students are eligible for the Summer Learning program if they received an ‘Incomplete’ in Math, Reading, or both during the Spring 2020 academic semester. As part of the Summer Learning Program, CPS is utilizing the online Khan Academy platform, a free, computer-assisted learning (CAL) programs available to teachers, students, and parents to support students in mathematics. The program is designed to save teachers’ time, lower education costs, and offer a more customizable learning experience to students. The goal is to facilitate interactive learning while allowing students to progress at their own pace. Such a readily available, no-cost platform makes it a promising tool for CPS to scale district-wide.

However, for students to benefit from the Summer Learning program and its curriculum, they first must be equipped with the proper information, technology, and face-to-face support to not only register but successfully participate in this virtual program.

INTERVENTIONS

We focus on three primary questions for this program evaluation:

1. Do personal outreach efforts to families induce registration for Summer Learning?
2. Do text message reminders increase attendance in Summer Learning?
3. Do personalized supports such as attendance monitoring, outreach to families, and individualized tutoring sessions increase attendance and performance in Summer Learning?

In addition, we are interested in the following secondary research questions:
4. What is the synergistic effect, if any, of getting both the text message reminders and the personalized supports during the Summer Learning program?
5. For this population of students, do additional supports during a four-week summer learning program influence their engagement with online learning in the following school year?

We plan to answer these questions by conducting three large-scale Randomized Control Trials (RCTs).

In the first RCT (referred to as Experiment 1 hereafter), we will randomize students eligible for the Summer Learning program who do not register by CPS’ registration deadline to two treatment arms: outreach worker treatment and control group. This effort will provide CPS with the opportunity to better understand whether they can improve program take-up through personal outreach. For students randomly assigned to the outreach worker treatment, outreach workers will reach out to students’ families using existing CPS contact information during the two weeks prior to the Summer Learning program. During such outreach, outreach workers will inform families of the Summer Learning program of their child’s eligibility to participate in the program; answer any questions they may have about the program; collect information on student barriers to virtual learning and/or any hesitations for registering for the program; and assist them with filling out the registration form. The control group will consist of randomly assigned students who will not be reached out to by outreach workers after the registration deadline is passed.

In the second RCT (referred to as Experiment 2 hereafter), which takes place while Summer Learning program is in session, we will randomize students who have registered for the Summer Learning program and consented to receive text messages into two treatment arms: text message reminders and control group. The text message reminders will be sent to parents to remind them of Summer Learning class times and to encourage them to have their child join their Summer Learning sessions.

In the third RCT (referred to as Experiment 3 hereafter), which also takes place during the Summer Learning program, we will randomize Summer Learning classrooms into two treatment arms: resident teacher support and control group. Resident teachers, who are teachers-in-training employed by CPS, will be randomly assigned to Summer Learning classrooms and will provide support to lead classroom teachers to increase student engagement and attendance at daily Summer Learning sessions; act as a mentor/role model to students and develop positive relationships with students in their assigned class(es); and support students to improve the use and efficacy of their online math learning platform – Khan Academy.

Experiment 2 and Experiment 3 start after completion of Experiment 1 and set up a 2x2 randomization scheme where students may receive the Resident Teacher treatment only, the texting treatment only, the Resident Teacher and texting treatment, or no treatment (control).

Experimental design

Experiment 1: We will evaluate the impact of outreach workers on Summer Learning registration using a randomized controlled trial. Students who were eligible for CPS’ Summer Learning program were randomized into either the outreach worker treatment group or the control group. Each of the 20 outreach workers received a random list of 400 students and proceeded to call parents working down their lists over the course of one week.

Experiment 2: We will evaluate the impact of text message reminders on Summer Learning participation and academic outcomes. Students whose parents registered them for Summer Learning and consented to receive text messages from CPS were randomized into either the text message reminder treatment or the control group. 3,804 students were randomized into the treatment group and 3,804 were randomized into control.

Experiment 3: We will evaluate the impact of classroom support in the form of a Resident Teacher on student Summer Learning participation and academic outcomes. We randomized Resident Teacher support at the classroom level. Out of 461 Summer Learning classrooms, 120 were randomly assigned Resident Teacher support.

Experiment 2 and Experiment 3 set up a 2x2 randomization scheme where students may receive the Resident Teacher treatment only, the texting treatment only, the Resident Teacher and texting treatment, or no treatment (control).

Design details

Randomization method
Experiment 1: Students are randomized at the individual level using Stata. We first remove two groups of students from our sample: those students who are already registered at the time of randomization those for whom we do not have a phone number listed in our data. Next, we randomly order and select 8,000 students from this sample to be in the treatment group and assign all the remaining students in our sample to the control group. We then check balance across treatment and control groups on the following baseline covariates: race, age, gender, English Language Learner status, students in temporary living situations status, learning disability, attendance days, member days, whether the student is eligible for free/reduced lunch, GPA, math and reading course failures, standardized math score, standardized reading score, and grade. We randomly sort the 8,000 treatment students into 20 individual lists for outreach workers, and check balance on those same variables across the 20 lists. Experiment 2: Students in Summer Learning were randomly assigned to the texting treatment at the individual level using Stata. We use CPS’s registration responses from the form they sent out to families to do this. We set a consistent seed, remove those students who are ineligible for Summer Learning (not in grades 1-8), remove those students without phone numbers available, and only keep those students who consented to be part of the study. We randomly assign 3,804 students (half of those who met the listed criteria) to the texting treatment. Experiment 3: The randomization for Experiment 3 is at the classroom level. Resident teachers are randomly assigned to Summer Learning classrooms using Stata. We randomize classrooms proportionally by grade band (1-3 grade, 4-5 grade, 6-8 grade) and by instructional language in the classroom (English or Spanish). Proportionally refers to allocating Resident Teachers (RTs) based on how many of each type of classroom in each grade-level there are in total for the Summer Learning program. We randomize only Resident Teachers with Spanish-speaking abilities to classrooms that were listed as a Spanish-speaking classroom. We assigned each resident teacher to support two classrooms. Each 1-5th grade lead teacher was only assigned to one classroom since they would teach the same group of students both math and reading; while each 6-8th grade lead teacher was assigned to teach only one subject (either math or reading) to two classrooms, so a math teacher would teach to two different groups of students. Given this structure and to fulfill the goal of assigning each Resident Teacher to support two classrooms, in 1st to 5th grade, we paired each Resident Teachers to support two lead teachers (and therefore two classrooms), while in 6-8th grade, we paired each Resident Teachers to support one math lead teachers (and therefore provide support for both of their classrooms). Since a key component of the resident teacher treatment was to provide support with engaging students with the online math platform, we did not assign resident teacher support to 6-8th grade reading lead teachers and assigned them to only support math teachers.
Randomization unit
Experiment 1: students are randomized at the student level for the registration experiment Experiment 2: The text message nudges are randomized at the student level. Experiment 3: The Resident Teacher treatment is randomized at the classroom level.
Clusters
Experiments 1 and 2 were not clustered. Experiment 3 was clustered at the classroom level, with 461 total classrooms.
Observations
Experiment 1: 14,567 total students Experiment 2: 7,608 total students Experiment 3: At the time of designing the experiment and writing up the pre-analysis plan, we do not have classroom roster data and therefore do not have any information on how students were distributed across classrooms/teachers. Consequently, we do not know the total number of observations for experiment 3.
Arms
Experiment 1: 8,000 students outreach worker treatment; 6,567 control students Experiment 2: 3,804 students receiving texts; 3,804 control students Experiment 3: 120 treatment classrooms; 341 control classrooms.
Primary outcomes
Our primary outcome variable for the registration experiment (Experiment 1) is registration in the Summer Learning program, which will be provided by CPS using their Aspen student information system. Our primary outcomes for Experiment 2 and 3 include: - Attendance during the Summer Learning program - Engagement with Khan Academy during the Summer Learning program
Secondary outcomes
Our secondary outcomes for Experiments 2 and 3 include: - Other measures of engagement in Summer Learning, including amount of time logged into Google Classroom, Summer Learning course completion, and end-of-program assessment scores, if available - Other measures of engagement with Khan Academy, including time spent on the Khan platform, number of problems attempted on Khan, and % mastered in the corresponding ‘getting ready for grade X’ course for students for whom Khan was used as their math online platform (grades 3-8), if available - Engagement with other online learning platforms such as ST Math and Amplify used by CPS during the summer learning program and the following school year (if available). - Academic outcomes during the following year as measured by: standardized math and reading test scores; math and reading course grades, overall GPA and Math GPA, course completion, and engagement with Khan during the following school year (SY2021). - Engagement with remote learning during the following school year

Cited by

No citation data available for this registration yet.

Keywords