Learning MORE (Model of Reading Engagement) Science and Social Studies Content and Vocabulary to Improve Third-Graders' Reading Comprehension at Scale: A Randomized Controlled Trial

How to cite

kim, james. 2021. "Learning MORE (Model of Reading Engagement) Science and Social Studies Content and Vocabulary to Improve Third-Graders' Reading Comprehension at Scale: A Randomized Controlled Trial." AEA RCT Registry. AEARCTR-0008337. https://doi.org/10.1257/rct.8337-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

james kim*

* Primary investigator

Registry ID
AEARCTR-0008337
DOI
https://doi.org/10.1257/rct.8337-1.0
First registered
Last updated
Status
In development
Trial dates
2021-10-06 – 2023-08-01
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/8337

Abstract

With a randomized field trial involving 112 Kindergarten to grade 5 elementary schools (N = 9,844 third-graders), we test the effectiveness of the MORE (Model of Reading Engagement) science and social studies intervention. We examine the hypothesis that building domain and topic knowledge schemas, through thematic content literacy instruction and a personalized literacy app, could improve third-graders’ ability to comprehend science and social studies passages. The literacy app is designed to improve students’ knowledge of words related to the themes in the teacher-directed lessons and to build students’ meta-linguistic skills. The study is implemented at scale in a southeastern US urban school district. Schools are first grouped into randomization blocks based on prior experience with the intervention, school size, and terciles of baseline reading scores and then randomly assigned to a control group which implemented science lessons and a literacy app with science content only or a treatment group which implemented science lessons and literacy app with science and social studies content and vocabulary. Students will be administered transfer measures of domain specific (science and social studies) reading comprehension, background knowledge, and reading motivation. Digital data from the literacy app will generate information on measures of behavioral engagement (total unique logins, total time on task, total activities completed) and cognitive engagement (average accuracy across all digital activities). We will conduct both confirmatory and exploratory analyses to estimate intent-to-treat impacts, meditational effects, and moderator effects. Our first confirmatory aim is to fit multi-level models nesting students within classrooms, and classrooms within schools, to generate intent-to-treat estimates of impact on transfer measures of domain specific reading comprehension. In addition, to estimate intent-to-treat estimates of impact on growth in reading, we will fit piecewise growth models across three time points to assess fall to spring growth on a vertically scaled, computer adaptive, domain general reading comprehension measure. Second, we will conduct mediational analyses to examine whether effects on reading comprehension outcomes are mediated via improvements in the vocabulary, background knowledge, reading motivation, and cognitive and behavioral engagement with a literacy application. Finally, we will examine whether treatment effects on transfer measures of reading comprehension are moderated by prior reading level (both baseline beginning of grade 3 reading), and school attendance from grade 1 to grade 3.

Intervention

To read for understanding and to prepare for college, young children must acquire the domain and topic knowledge needed to read complex texts. Despite the growing literacy demands in the 21st century, fewer than 5% of US schoolchildren in the elementary grades can evaluate complex nonfiction texts with high background knowledge and vocabulary demands. Furthermore, there are larger gaps based on family income and student ethnicity on reading comprehension outcomes rather than basic word reading skills (D’Agostino & Rodgers, 2017; Fryer & Levitt, 2006; Reardon et al., 2012). During COVID-19, there is also increasing evidence that gaps in early grade reading comprehension tests have grown wider (Kuhfeld et al., 2021). Importantly, recent evidence suggests that the over-emphasis on basic literacy and math skills have contributed negative effects on science and social studies texts and exacerbated achievement disparities based on students’ racial and ethnic background, as well as their socioeconomic status (Arold & Shakeel, 2021; Song et al., 2019).

In many ways, US schools are increasingly focused on technology-based efforts to quickly personalize remediation efforts in literacy and math to help recover students' learning losses during COVID-19 (TNTP, 2021). At the same time, many states are enacting policies designed to improve K-3 literacy through ongoing investments in instructional approaches aligned with the science of reading (Burk, 2020; Folsom et al., 2017). Thus, there is a great need to identify evidence-driven literacy interventions that integrate teacher-directed instruction with educational technology that can help build students’ vocabulary and content knowledge while simultaneously providing more targeted and personalized activities to build more foundational literacy skills.

In response to this urgent national challenge, we designed this clustered randomized controlled trial to evaluate the replicability and scalability of the Model of Reading Engagement (MORE) content literacy intervention. In essence, the MORE intervention lessons and literacy apps are designed to build students’ domain knowledge and promote engagement with thematically related vocabulary (Connor et al., 2017; Guthrie & Klauda, 2014; Romance & Vitale, 2001; Vaughn et al., 2013; Williams et al., 2016). It also provides children with digital activities through a literacy app that supports the development of meta-linguistic awareness (MA). Researchers have defined MA as a child’s ability to analyze and play with language as an object independent from its meaning (Cazden, 1974; Roth et al., 1996; Tunmer & Bowey, 1984). MA is multi-dimensional and includes phonological and morphological awareness, syntactic awareness, and the ability to resolve semantic ambiguities in figurative language, such as riddles, jokes, and puns (Cairns et al., 2004; Yuill, 1996). There is emerging experimental evidence that brief interventions focused on building MA can support improvements in elementary grade students’ reading comprehension outcomes (Zipke, 2008; Yuill, 1996, 2009).

In this study, we developed a literacy app that included games to help students build their MA as they played games to build their knowledge of domain-specific science and social studies words and appropriately challenging single syllable words. In a previous validation study, we found that student performance, as measured by accuracy on the literacy app games, predicted improvement on both domain-specific and domain-general measures of reading comprehension, controlling for student covariates such as prior reading and math skills, demographic characteristics, and school and neighborhood contexts. These analyses found that a one standard deviation increase in accuracy predicted a 0.10 improvement in Spring formative assessments and a 0.15 increase in the North Carolina End of Grade exam. The magnitude of the association between cognitive engagement with the literacy app and student outcomes is consistent with recent meta-analyses of educational apps in literacy and math that show mean effect sizes of approximately 0.11 standard deviations on unconstrained skills like vocabulary and comprehension (Kim et al., 2020).

Experimental design

Researchers at the READS Lab at the Harvard Graduate School of Education will conduct a cluster randomized controlled trial. We stratified at the school level by prior experience with MORE lessons, student 3rd grade enrollment, and Fall achievement on the North Carolina Beginning of Grade (NC BOG) assessment in reading. For schools that had prior experience with MORE, there are five schools in each block, while for schools with no prior experience with MORE there are fourteen schools in each block. The treatment is clustered at the school-level. Next, we randomly assign whether the schools will receive treatment within each block. For blocks with an odd number of schools we randomize whether the first school is treatment or control.

Design details

Randomization method
Randomization was conducted in an office by a computer and implemented using STATA code.
Randomization unit
The unit of randomization is the school, blocked by prior experience with MORE, 3rd grade enrollment size, and performance on the NC BOG during the 2021-22 school year.
Clusters
112 school clusters
Observations
All students in the district will participate in the intervention. As of September, there were 9,844 3rd grade students enrolled in the district. We anticipate that 5% of the students will exit the district prior to the end of year assessment yielding a sample of 9,350.
Arms
56 Schools with 4,890 students in the Science-only MORE and 56 Schools with 4,954 students in the Science and Social Studies MORE
Primary outcomes
There are three primary outcomes. First, we will administer a researcher-developed measures of students’ domain specific reading comprehension in science and social studies, vocabulary depth, background knowledge, and reading motivation. Second, we will use administrative data on two domain general reading comprehension assessments - the state End of Grade Exam and and vertically scaled and computer adaptive Northwest Evaluation Association's Measure of Academic Progress. Finally, a measure of cognitive and behavioral engagement assessed in the literacy app.
Secondary outcomes
There are two secondary measures of fidelity: a teacher survey of knowledge and practices and teacher discourse during read alouds.

Cited by

No citation data available for this registration yet.

Keywords