Elevating Innovation in Teaching and Learning: Understanding the Impact of a Whole-School Personalized Learning Model in CPS

How to cite

Khan, Salman; Momeni, Fatemeh; Bhatt, Monica; Guryan, Jonathan. 2022. "Elevating Innovation in Teaching and Learning: Understanding the Impact of a Whole-School Personalized Learning Model in CPS." AEA RCT Registry. AEARCTR-0009097. https://doi.org/10.1257/rct.9097-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Salman Khan* , Fatemeh Momeni , Monica Bhatt , Jonathan Guryan

* Primary investigator

Registry ID
AEARCTR-0009097
DOI
https://doi.org/10.1257/rct.9097-1.0
First registered
Last updated
Status
On going
Trial dates
2017-07-06 – 2024-09-01
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/9097

Abstract

Reducing disparities in schooling outcomes between low-income youth and their more affluent peers is a top policy priority in the United States (US). Improving schooling outcomes is central to addressing a wide range of social problems including disparities in poverty (Goldin & Katz, 2008), health outcomes (Lleras-Muney, 2005), and criminal justice involvement (Lochner & Moretti, 2004). The public education system in the US is one of the most important formal mechanisms society has for developing human potential in youth. Responsible for educating 90% of all children, the public education system has the capability to level the playing field among children of different backgrounds, starting at an early age (Hussar et al., 2018; McFarland et al., 2020). The extent to which schools can effectively achieve this aim in equity lies in their ability to cultivate and maintain a conducive learning environment that works for all students.

One promising avenue to improve educational outcomes for all students is to implement personalized learning (PL) techniques, which changes the core foundations of teaching and learning. PL refers to giving more individualized instruction to students so that they can take ownership over the pace and environment of their learning. Education researchers have understood that individualizing instruction has the potential to generate “the best learning conditions we can devise” (Bloom, 1984). This approach allows students who have different learning styles, backgrounds, and abilities to all get the level of attention they need to academically succeed. The PL approach posits that there may be a fundamental “mismatch” between what many students need – particularly those students whose backgrounds and early life experiences differ from those of majority populations – and what regular classroom environments deliver. Instead of facilitating individualized learning environments in schools, providing resources, and piloting innovative ways to systematically personalize instruction; education systems have asked teachers to instruct large classrooms of children with varied skills and needs – an approach that ignores contextual factors of students that we know are important for effective learning.

The intervention we study here – the Elevate model of PL being piloted by Chicago Public School’s (CPS) Department of Personalized Learning – attempts to comprehensively transform the learning process through a personalized approach, which is centered on technology adoption and social and emotional learning (SEL) support. Elevate is a learner-driven instructional model that fosters 21st century skills by empowering every student to actively co-design their learning path, pace, and environment according to their individual needs, strengths, and interests. In this flexible learning model, learning can happen anytime and anywhere, and students are encouraged to demonstrate their knowledge in multiple ways.

To better understand the effects of PL as a whole-school initiative, Chicago Public School’s (CPS) launched Elevate. Elevate attempts to comprehensively transform the learning process through a personalized approach, which is centered on technology adoption and social and emotional learning (SEL) support. The program is designed to better support students in the district by providing them with the individualized instruction that they need to succeed at the school level. Elevate revolves around three main components that have been shown to be key elements of successful PL initiatives: (i) Professional Development (PD) supports to administration and teachers to prepare them with the necessary PL teaching skills and create an aligned vision with students (ii) ed-tech learning programs that have consistently shown a positive impact on instructional learning activities for students and (iii) funding for classroom redesign that has been proven to be a more positive environment for student learning (Roschelle et al., 2010; Zhang et al., 2020; Hammerschmidt-Snidarich et al., 2019). In an effort to understand the implementation and impact of this significant public investment in whole-school PL, CPS Personalized Learning Department (PLD) partnered with the University of Chicago Education Lab to evaluate the Elevate model.

Intervention

Background

Elevate is designed to be fully implemented in a given school over the course of 2.5 years. Schools participating in Elevate receive a variety of services to adopt the PL environment including: intensive PD support for teachers and administrators from LEAP Innovations (LEAP ), additional compensation for teacher training time, technology devices to achieve 1:1 device-student ratios, ed-tech programs (ex: ALEKS), flexible learning environment enhancements including furniture and classroom visual improvements, and access to both a CPS Instructional Coach and a CPS SEL Coach to support teachers and administrators transitioning the school into a fully implemented PL model. These services are explained in detail below.

Elevate PL Intervention Program Components

Professional Development

LEAP serves as the lead PD provider for teachers and administrators in the Elevate program. Schools receive support from LEAP through the 2.5 years as they embark on a holistic redesign of their school. The focused set of full day PD sessions, provided by LEAP, takes participating teachers through PL in action, strategies for managing the change, the role of technology in supporting PL, planning for effective pilots through continuous improvement cycles, and selection of technology tools to support the success of the PL in the classroom.

LEAP is a leader in personalized learning and has helped CPS introduce PL in schools since 2014. LEAP has had previous partnerships with CPS through the LEAP Pilot Network and the Breakthrough Schools: Chicago (LEAP, 2017).

Instructional and SEL Coaches

Elevate assigns two Instructional Coaches and one SEL Coach to support participating schools in their transition. The Instructional Coach supports teachers in developing strategies with individualized student-driven academic teaching and the SEL Coach supports teachers in implementing individualized culture, student reflection, and emotional teaching. Coaches are housed within CPS PLD but work in schools four or five days per week. They collaborate regularly with LEAP, as well as staff at each school. Though the PD program with LEAP ends in December of the third school year (as explained further below), Coaches will remain at their respective CPS schools through the end of second semester of the third school year to ensure that implementation plans are seamlessly continued. LEAP also provides additional coaching support to pilot teachers in Year 2 of the program.

Technology – Devices and Software

Participating schools receive devices in order to achieve a 1:1 student-to-device ratio. The type of device is determined by school need and in collaboration with the Instructional Coach. Up to three PL software programs per student are available in each classroom. Devices and software programs are delivered to classrooms as soon as teachers start incorporating PL in their teaching methods in Year 2 of the program. An example ed-tech program is ALEKS (Assessment and Learning in Knowledge Spaces), a web-based interactive learning system that adapts to student knowledge to ensure various topics are learned and retained (McGraw Hill, n.d.).

Classroom Redesign – Furniture and Facility Enhancements

21st century classroom spaces allow for flexible seating and student-centered learning spaces that meet SEL and academic needs. New furniture in all classrooms and capital improvements in the classrooms are core investments in the whole-school redesign program. Classrooms receive furniture and facility enhancements, such as paint or floor repair, as soon as a teacher is onboarded to implement PL in their classroom.

Experimental design

We evaluate the impact of Elevate on student outcomes using a clustered RCT design, where we randomly selected the eligible schools to either receive the intervention or be assigned to a control group. To track the implementation of various components of Elevate and document variability in program implementation we partnered with the American Institutes for Research (AIR) to conduct a 4-year implementation study from SY18–19 to SY21–22.

Randomized Control Trial

To answer our research questions, we implemented the Elevate program as a clustered RCT, in partnership with CPS. CPS identified 45 eligible elementary schools. The eligible schools were selected based on the criteria listed in Table 2. Each eligible school was sent an invitation by the CPS PLD to complete an interest survey. All 45 invited schools successfully completed the interested survey. Once all schools completed the interest survey, we randomized them into treatment and control conditions over two cohorts, using a school-level cluster randomized design. Within each cohort, eligible schools were separated into two groups, with or without prior PL exposure, and then randomly assigned a condition. In Cohort 1, out of the 21 eligible schools that expressed interest in participating, 10 were assigned to the treatment group and 11 were assigned to the control group. In Cohort 2, out of the 24 schools that expressed interest in participating, 16 were assigned to the treatment group and 8 were assigned to the control group.

(Table 2 can be found on pg. 1 in the attached supporting materials)

Schools that were assigned to the treatment condition then moved forward with submitting an application. All schools that were randomized into treatment were invited by CPS PLD to fill out an application form. All schools opted to complete the application to qualify for the program, but CPS decided to exclude some schools from receiving the program as they did not have the necessary infrastructure in place to implement the program successfully. Then a final group of principals attended a site visit to a current PL school, where they received a full overview of the Elevate program. Principals were encouraged to speak with their staff before making a final decision to opt in or out of the program. The final allocation of Elevate programming was determined by which principals agreed to committing to the PL practices of Elevate after this site visit. Crucially, in our evaluation, we estimate both intent-to-treat (ITT) estimates using random assignment in addition to treatment-on-the-treated (TOT) estimates with respect to which schools decided to implement the program. The intent-to-treat estimates will be based on initial randomization of treatment assignment.

Implementation Study

From our experiences in Chicago, we have learned that it is critical to have a deep understanding of what is happening in the schools in order to ensure that the program is implemented with fidelity to the research design. As such, we partnered with the American Institutes for Research (AIR) to use a structured interview protocol that documents key aspects of program implementation (such as district-level investment and infrastructure; school-level investment and infrastructure; PD; coaching: teachers; coaching: leadership; and deliverables/results). The implementation fidelity study by AIR classified schools into high, medium, and low implementing groups. As a part of the evaluation plan, the research team will explore whether the schools that benefit the most from Elevate are those that are identified by AIR as high implementers.

Design details

Randomization method
Randomization was completed using Stata, a statistical software package. Within each cohort and strata (schools with and without prior PL history), schools were randomized by 1) assigning each school a random number from a uniform distribution between 0 and 1; 2) ordering the list of schools from smallest to biggest number; and 3) selecting the top N schools on the list for treatment, where N varied based on the size of the randomization block and number of treatment opportunities.
Randomization unit
We implemented the Elevate program as a clustered RCT across elementary schools. Randomization occurred on the school-level. Schools were randomized within cohort and within strata according to whether they had a prior history of personalized learning (4 randomization blocks).
Clusters
45 schools across two cohorts (26 assigned to treatment and 19 assigned to control)
Observations
At the time of randomization, a total of 22,227 students were enrolled across all 45 study schools. Our analysis will include observations from students who will be enrolled in grades 3-8 at the study schools a year and three months after the end of the 2.5-year program and who have attended that school for at least one day that school year. We may drop randomization blocks that do not have enough data to make a comparison between treatment and control schools. We foresee this might be an issue since the two randomization blocks of schools with a history of personalized learning only contain 2 schools each.
Arms
26 schools were assigned to treatment and 19 were assigned to control: Cohort 1: 10 treatment schools, 11 control schools Cohort 2: 16 treatment schools, 8 control schools
Primary outcomes
The primary research question for this study will be: 1. What is the impact of Elevate on student academic performance, as measured by combined standardized reading and math test scores one year after (intended) program completion date? This approach will allow us to closely approximate our initial plan to study primary outcomes during the last year of the program. However, due to the COVID-19 pandemic, CPS cancelled the End of Year (EOY) exams for SY 19-20 and SY 20-21. Under the revised timeline, this outcome will be studied using the beginning of year (BOY) Star 360 test scores for School Year (SY) 21-22 for Cohort 1 and BOY SY 22-23 Star 360 test scores for Cohort 2.
Secondary outcomes
We will conduct the following secondary/exploratory analyses, including investigating the impact of Elevate on other student-level (academic and SEL) and teacher-level (i.e. teacher and principal retention) outcomes, as well as effects on school composition, in addition to conducting various subgroup analyses. Student Level Academic Outcomes 2. What is the impact of Elevate on standardized reading and math test scores separately one year after (intended) program completion date, using BOY Star 360 test scores? 3. What is the impact of Elevate on standardized test scores during the final year of program implementation? This approach will use standardize test scores such as NWEA, IAR, and Star 360. 4. What is the impact of Elevate on standardized test scores one year after program completion date using standardized tests other than STAR 360, depending on availability. 5. What is the impact of Elevate on GPA, number of courses failed, and drop-out rates in the year after program completion? Student SEL Outcomes & School Climate 6. What is the impact of Elevate on number of days attended in the year after program completion? 7. What is the impact of Elevate on student SEL and behavioral outcomes, as measured by the number of misconduct incidents and number of suspensions per student in the year after program completion? 8. What are the impacts of Elevate on measures of school climate as measured by student and teacher responses to My Voice, My School (MVMS) self-reported survey one year after the final year of the program implementation? Teacher/Admin Level Outcomes 9. What is the impact of Elevate on teacher retention as measured by the share of initial teachers at a school who are still teaching at the same school by one year after the end of the program implementation? 10. What is the impact of Elevate on principal retention as measured by the share of initial principals at a school who are remain at the same school by one year after the end of the program implementation? School Composition 11. What is the impact of Elevate on student and teacher composition at the school-level one year after the final year of the program implementation? a) Are demographic characteristics of students at Elevate schools (such as racial and gender composition, share of students eligible for free or reduced-price lunch, share of students who have an individualized learning program plan, and share of English Learner students) different from those in control schools? b) Are professional characteristics of teachers (as measured by average years of experience and highest degree earned) at Elevate schools different from those in control schools? Spillover Effects 12. Are there spillover effects within Elevate schools from teachers who are trained earlier to teachers who are trained later as measured by student academic outcomes including standardized test scores, GPA, course failure, and attendance? a) Are there spillover effects from pilot teachers to non-pilot stage teachers? b) Are there spillover effects from stage 1 and pilot teachers to stage 2 teachers? Sub-Group Analysis 13. Are students from different demographic groups (e.g., race, gender, age, etc.) impacted differently by Elevate? 14. We will also check the robustness of our primary findings by examining each cohort separately.

Cited by

No citation data available for this registration yet.

Keywords