Scaling Effective Programs: Testing Strategies of Scale in the Context of a Larger Randomized Controlled Trial
How to cite
Scherer, Ethan. 2026. "Scaling Effective Programs: Testing Strategies of Scale in the Context of a Larger Randomized Controlled Trial." AEA RCT Registry. AEARCTR-0019252. https://doi.org/10.1257/rct.19252-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
In many sectors, identifying effective program through randomized controlled trials is rare, but even rarer are effective program that scale to multiple sites and replicate the original study's findings and magnitudes. Part of the reason for this result is that in the social sciences, adoption and diffusion of an innovation involves a web of intersecting factors that sociologist Everett Rogers articulated more than 80 years ago (Rogers, 2003). Yet across sectors, achieving consistent take-up is still rare. Thus, to reverse the deep learning recession of more than a decade (Mahnken, 2026), education will need more than rigorous research on effect programs, it will also need to identify and test scaling components that lead to broad and deep adoption of these program.
This project aims to explore the effectiveness of two connected strategies to scale an evidence-based program to 19 schools in a western state district. It leverages insights from Rogers original work in the health sector and applies them to the education context. The schools are part of a larger randomized controlled trial to assess the efficacy of an elementary science and social studies program that has led to both literacy and mathematics gains in two prior trials (Kim et al., 2021l Kim et al., 2024). Within this context we test two strategies. First we randomly assign schools whether they can select a teacher innovator. Teacher innovators have the responsibilities to try, experiment, and fail with with implementing the program, but have the responsibility to share these findings with other teachers, coaches and district officials, so others can learn from their experience. A second strategy will be to experiment with with the timing of district-level communication - a top-down approach. The randomized timing of the messages will assess whether certain message content is more effective than others at improving take-up and usage.
We will explicitly looks at the effect of each strategy on implementation outcomes, but also whether these improvements in implementation lead to improvements in student outcomes in the context of the larger randomized controlled trial.
Intervention
The Model of Reading Engagement scaling strategy focuses on three levers of change involving (1) the Summer Leadership Institute (SLI), (2) a district-wide tiered communication and implementation plan, and (3) MORE Teacher Innovators (TIs) that help build district capacity and buy-in at multiple levels of the system.
At the Summer Leadership Institute (SLI), key systems leaders (e.g., Chief Academic Officer, responsible district staff) to understand what MORE is, an elementary science and social studies program, its impacts on student learning, how it helps to accomplish the agency’s strategic goals, and to develop a plan on how to communicate their learning and support to school and district employees throughout the year. In additional school leaders and teacher innovators attend the Summer Leadership Institute. Teacher Innovators are selected by the school principal and willing to experiment with MORE tools.
The district tiered communication plan leverages real-time implementation data collected by the MORE program. At SLI clear benchmark (e.g., mile markers) are identified and communicated to all stakeholders. Then the tiered communication focuses four types of messages: 1) informational which all participants receive, 2) celebrations of success, celebrating stakeholders who met our exceeded the implementation benchmarks with specific and concrete data embedded into the message, 3) reminders to those who are not meeting the benchmarks with specific and concrete data embedded into the message, and 4) learning from the teacher innovators experimentation. Message types (2) and (3) are informed research that specific and targeted information are effective (Asher, Scherer, & Kim, 2022) These four types of automated messages are sent to school coaches first, followed by the principal, and then finally data are reviewed during a virtual data meeting with district stakeholders to decide on more personalized adjustments. The timing of these messages has been randomized to assess incremental improvements in implementations.
Finally, TIs goal is to assist in identifying logistical issues before the tools are rolled out more broadly and experiment with MORE and its principles. TIs will conduct a plan-do-study-act (PDSA) cycles and attend several 1-hour coaching session from the MORE team. They then will continue to conduct these PDSA cycles with coaching from the MORE Team early in the year up to four times. They have the responsibility to then disseminate what they learned to other teachers in their school and to district leaders.
The randomized controlled trial aims to test the efficacy for the messages on implementation as well as the effect of teacher innovators on the MORE implementation data as well as student outcomes. This is taking place in the context of a larger randomized controlled trial to assess the efficacy of MORE.
Experimental design
The study will employ a blocked cluster (school) RCT design. All participating schools will take part in MORE and we will randomly vary which grade levels receive the treatment. Specifically, half of the participating schools will implement the MORE treatment in grade 1 (Group A) and the other half will implement MORE in grade 3 (Group B). Treatment in this case means they receive all three MORE tools (e.g., digital activities, lessons, and formative assessment) while the control means that they only receive two of the MORE tools (e.g., digital activities and formative assessment). This approach allows each school to contribute both to the treatment and business-as-usual (BAU) control sample. In those schools randomly assigned to implement MORE in grade 1 (Group A), grade 3 will serve as the BAU control group for Group B, which implements MORE in grade 3. Similarly, in those schools randomly assigned to implement MORE in grade 3 (Group B), grade 1 will serve as the BAU control group for Group A.
Within the treatment assignment, treatment grades were randomly assigned to receive a MORE Teacher Innovator or not. In cases where schools did not receive a teacher innovator, they will be assigned to receive the first tiered message. Control schools-grades will receive the same message a day or two after the treatment grades.
Design details
Cited by
No citation data available for this registration yet.