Scaling Effective Programs: Testing Strategies of Scale in the Context of a Larger Randomized Controlled Trial

How to cite

Scherer, Ethan. 2026. "Scaling Effective Programs: Testing Strategies of Scale in the Context of a Larger Randomized Controlled Trial." AEA RCT Registry. AEARCTR-0019252. https://doi.org/10.1257/rct.19252-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Ethan Scherer*

* Primary investigator

Registry ID
AEARCTR-0019252
DOI
https://doi.org/10.1257/rct.19252-1.0
First registered
Last updated
Status
In development
Trial dates
2026-08-18 – 2028-12-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/19252

Abstract

In many sectors, identifying effective program through randomized controlled trials is rare, but even rarer are effective program that scale to multiple sites and replicate the original study's findings and magnitudes. Part of the reason for this result is that in the social sciences, adoption and diffusion of an innovation involves a web of intersecting factors that sociologist Everett Rogers articulated more than 80 years ago (Rogers, 2003). Yet across sectors, achieving consistent take-up is still rare. Thus, to reverse the deep learning recession of more than a decade (Mahnken, 2026), education will need more than rigorous research on effect programs, it will also need to identify and test scaling components that lead to broad and deep adoption of these program.

This project aims to explore the effectiveness of two connected strategies to scale an evidence-based program to 19 schools in a western state district. It leverages insights from Rogers original work in the health sector and applies them to the education context. The schools are part of a larger randomized controlled trial to assess the efficacy of an elementary science and social studies program that has led to both literacy and mathematics gains in two prior trials (Kim et al., 2021l Kim et al., 2024). Within this context we test two strategies. First we randomly assign schools whether they can select a teacher innovator. Teacher innovators have the responsibilities to try, experiment, and fail with with implementing the program, but have the responsibility to share these findings with other teachers, coaches and district officials, so others can learn from their experience. A second strategy will be to experiment with with the timing of district-level communication - a top-down approach. The randomized timing of the messages will assess whether certain message content is more effective than others at improving take-up and usage.

We will explicitly looks at the effect of each strategy on implementation outcomes, but also whether these improvements in implementation lead to improvements in student outcomes in the context of the larger randomized controlled trial.

Intervention

The Model of Reading Engagement scaling strategy focuses on three levers of change involving (1) the Summer Leadership Institute (SLI), (2) a district-wide tiered communication and implementation plan, and (3) MORE Teacher Innovators (TIs) that help build district capacity and buy-in at multiple levels of the system.

At the Summer Leadership Institute (SLI), key systems leaders (e.g., Chief Academic Officer, responsible district staff) to understand what MORE is, an elementary science and social studies program, its impacts on student learning, how it helps to accomplish the agency’s strategic goals, and to develop a plan on how to communicate their learning and support to school and district employees throughout the year. In additional school leaders and teacher innovators attend the Summer Leadership Institute. Teacher Innovators are selected by the school principal and willing to experiment with MORE tools.

The district tiered communication plan leverages real-time implementation data collected by the MORE program. At SLI clear benchmark (e.g., mile markers) are identified and communicated to all stakeholders. Then the tiered communication focuses four types of messages: 1) informational which all participants receive, 2) celebrations of success, celebrating stakeholders who met our exceeded the implementation benchmarks with specific and concrete data embedded into the message, 3) reminders to those who are not meeting the benchmarks with specific and concrete data embedded into the message, and 4) learning from the teacher innovators experimentation. Message types (2) and (3) are informed research that specific and targeted information are effective (Asher, Scherer, & Kim, 2022) These four types of automated messages are sent to school coaches first, followed by the principal, and then finally data are reviewed during a virtual data meeting with district stakeholders to decide on more personalized adjustments. The timing of these messages has been randomized to assess incremental improvements in implementations.

Finally, TIs goal is to assist in identifying logistical issues before the tools are rolled out more broadly and experiment with MORE and its principles. TIs will conduct a plan-do-study-act (PDSA) cycles and attend several 1-hour coaching session from the MORE team. They then will continue to conduct these PDSA cycles with coaching from the MORE Team early in the year up to four times. They have the responsibility to then disseminate what they learned to other teachers in their school and to district leaders.

The randomized controlled trial aims to test the efficacy for the messages on implementation as well as the effect of teacher innovators on the MORE implementation data as well as student outcomes. This is taking place in the context of a larger randomized controlled trial to assess the efficacy of MORE.

Experimental design

The study will employ a blocked cluster (school) RCT design. All participating schools will take part in MORE and we will randomly vary which grade levels receive the treatment. Specifically, half of the participating schools will implement the MORE treatment in grade 1 (Group A) and the other half will implement MORE in grade 3 (Group B). Treatment in this case means they receive all three MORE tools (e.g., digital activities, lessons, and formative assessment) while the control means that they only receive two of the MORE tools (e.g., digital activities and formative assessment). This approach allows each school to contribute both to the treatment and business-as-usual (BAU) control sample. In those schools randomly assigned to implement MORE in grade 1 (Group A), grade 3 will serve as the BAU control group for Group B, which implements MORE in grade 3. Similarly, in those schools randomly assigned to implement MORE in grade 3 (Group B), grade 1 will serve as the BAU control group for Group A.

Within the treatment assignment, treatment grades were randomly assigned to receive a MORE Teacher Innovator or not. In cases where schools did not receive a teacher innovator, they will be assigned to receive the first tiered message. Control schools-grades will receive the same message a day or two after the treatment grades.

Design details

Randomization method
Randomization done in office by a computer
Randomization unit
School-Grade
Clusters
38
Observations
2,200
Arms
The sample size by treatment arm is the same as the total number of clusters. All schools will participate in the treatment, with one grade (i.e., first, third) randomly assigned to treatment and the other randomly assigned to control. Treatment grades were assigned a teacher innovator. School who did not receive a teacher innovator will receive the first time messages.
Primary outcomes
There are six primary outcomes. Two measures of teacher engagement: teacher login and module completion. Two measures of students engagement: Student login and completion of one "world." Finally two proxy measures of completion of the lessons: completion of the science and social studies assessment.
Secondary outcomes
There are two secondary outcomes. First, we will administer a researcher-developed measures of student knowledge of science and social studies concepts taught in the MORE lessons. Second, we will use administrative data on a measure of early literacy skills (MCLASS Dynamic Indicators of Basic Early Literacy Skills (DIBELS).

Cited by

No citation data available for this registration yet.

Keywords