Impact Evaluation of TipsByText: A Parent-Oriented Language Program for Preschool Children

How to cite

Simonsen, Marianne; Loeb, Susanna; Lilleør, Helene Bie; Riis-Vestergaard, Michala Iben. 2020. "Impact Evaluation of TipsByText: A Parent-Oriented Language Program for Preschool Children." AEA RCT Registry. AEARCTR-0006393. https://doi.org/10.1257/rct.6393-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Marianne Simonsen* , Susanna Loeb , Helene Bie Lilleør , Michala Iben Riis-Vestergaard

* Primary investigator

Registry ID
AEARCTR-0006393
DOI
https://doi.org/10.1257/rct.6393-1.0
First registered
Last updated
Status
On going
Trial dates
2019-10-01 – 2022-12-31
Countries
Private
JEL codes
I240
Registry record
http://www.socialscienceregistry.org/trials/6393

Abstract

Early-life skills acquisition is a strong predictor of later life outcomes such as education and income, even in Denmark with highly subsidized childcare and free education. Parents play an instrumental role in their children’s acquisition of skills, but research suggests that parents from different socioeconomic backgrounds invest differently in their children. Consequentially, substantial skills gaps emerge between children from different family backgrounds already at an early age, and research suggests that these gaps can be difficult to close at later ages. In this paper, we evaluate the impact of a low-cost way to reduce the early skills gaps: by sending text messages directly to parents of preschool children with tips about how to incorporate simple language stimulation exercises into their busy everyday lives. The text messages are designed to eliminate some of the behavioral barriers to active language stimulation: lack of information, cognitive load, and discounting of future rewards. The intervention was originally developed by American researchers at Stanford University and has been adapted to the Danish context. The evaluation is designed as an RCT with a total of 3,181 families across five municipalities in Denmark.

Intervention

TipsByText is a text-messaging program that aims to help parents support their children’s language development in the preschool years. For eight months, parents receive three weekly text messages with tips for fun and simple language-stimulating games and activities they can do with their children. The text messages are formulated in non-complex language and the tone in the texts is short, direct, and with a call for immediate action in daily routines, e.g., bath time. The curriculum is aimed at parents of three- to five-year-old children, as children in Denmark typically enter preschool around the age of three and stay there until they start primary school (kindergarten class) in the year they turn six.

The intervention is an adapted version of an American parent education program, Ready4K, developed by Susanna Loeb and Benjamin York of Stanford University. The intervention is designed around three behavioral barriers to involved parenting: First, some parents have insufficient information about the impact their child-directed speech has on their child’s language development. To overcome this barrier, each Monday parents receive a “FACT” text designed to generate knowledge by highlighting the importance of particular skills. Second, parenting is a complex task and the cognitive load of parenting can result in a low prioritization of active language stimulation. To help alleviate this barrier, each Wednesday the parents receive a “TIP” text, which contains a specific and creative exercise on how to incorporate a language-stimulating element into a daily routine that is already taking place. Third, to motivate parents and keep them engaged in the language support of their children, each Friday the parents receive a “GROWTH” text, which provides encouragement and a deeper insight of the benefits of the exercise given on Wednesday.

The Danish adaptation of the intervention builds on the same principles as the American version but is specific to the Danish set-up and culture. In particular, the Danish curriculum focuses on three components of language development, which are a priority for language development in the public childcare sector. These components are:
• Formal language (vocabulary: understanding and producing words)
• Early literacy (letters, reading direction, etc.)
• Use of language (conversation, dialogue, narrativity, taking turns, and initiatives).
The adaptation was facilitated by the ROCKWOOL Foundation Interventions Unit (RFI) and was carried out in collaboration with Susanna Loeb and a diverse group of researchers and stakeholders who represent a wide knowledge of early language acquisition, child psychology, child development, and parent education programs and/or early interventions in Denmark. This group consisted of Pia Thomson, who is a prominent Danish language researcher, local municipalities, practitioners, and parent groups.

Experimental design

We employ a family-level randomized controlled trial with an opt-out design for parents of children aged three-to-five in five partnering municipalities in Denmark: Hoeje-Taastrup, Ikast-Brande, Langeland, Lejre, and Middelfart.

Recruitment: Each of the five municipalities provided us with a list of preschools that would participate in the evaluation; it was at the municipalities’ discretion to determine which preschools would participate, and the municipalities included between 67% and 100% of public preschools.

Sample: In total, 3,804 children born in or later than year 2016 (age 3 at the start of the intervention) were enrolled in the 70 participating preschools on October 1, 2019; these children constitute our population. Children in the population were included in the trial if two conditions were met on January 1st, 2020: First, that their parents had not opted out of the trial, second, that we had received at least one parental phone number from the collaborating municipalities. In total, 3,498 children in 3,181 families were included in the trial. At the start of the intervention their age distribution was as follows: 1,024 3-year-old children, 1,106 4-year-old children, 1,219 5-year-old children, and 149 6-year-old children.

Timing: Baseline language assessments were performed by the collaborating preschools in the Fall of 2019. Randomization was conducted in January 2020. On January 24, 2020, parents in the treatment condition received the first text message. Children in the control condition received treatment as usual. The last text message will be sent out on September 5, 2020.

Endline data: In September 2020, the municipalities will start collecting the first round of child-level endline data on language development (using “Sprogvurdering 3-6”). The data collection is scheduled to be complete by the end of 2020, at which time we will receive the data. In September-November 2021 and 2022, respectively, the municipalities will collect the second and third rounds of child-level endline data on language development (also using “Sprogvurdering 3-6”).

Information provided to parents: All parents in the participating preschools received the same informational pamphlet about the intervention in October 2019 before any data was collected. The pamphlet describes the intervention, the evaluation (including randomization), the data collection, and the research team. The information was delivered in printed form to the parents in the wardrobe of each preschool and was uploaded to most preschools’ intranet. The material was translated into eight languages; all parents received the printed version in Danish, and for each child the personnel in the preschool decided whether to attach a translated version. The material provided the parents with a link to a website that gave additional information, and with a phone number/email they could contact if they wished to opt out of the trial. In total, parents of 52 children opted out of the trial before randomization was conducted.
Spillover/contamination risk: To minimize spillover and contamination of the intervention to the control group, we aimed to minimize the probability that the preschool staff knew the allocation of children into the treatment or control group. We used three strategies to achieve this. First, the preschool staff was given no information about the allocation. Second, the preschool staff was explicitly instructed not to initiate conversations with the parents about the intervention. Third, to minimize the risk of parents engaging the preschool staff in conversation about the intervention, the informational pamphlet does not mention the preschool. Instead, the sender of the informational pamphlet is the municipality and the ROCKWOOL Foundation Interventions Unit.

Design details

Randomization method
Randomization of treatment status was done by a computer using the randtreat command in Stata 15.1. Families were randomized into three different groups: Treatment group, Control group, and Interview group. First, 119 families were drawn at random (stratified by municipality and whether the youngest child’s baseline language assessment was above or below the median) to be interviewed about their experience with the intervention and were removed from the RCT sample. Second, the remaining families were randomized 50/50 into treatment and control group stratified by three variables: 1. preschool (70 strata), 2. whether the youngest child scored above/below the median within their age group in the baseline language assessment or has a missing baseline language assessment (3 strata), and 3. expected language assessment material at endline (4 strata). In total there were 840 strata, and misfits were allocated using the global-method in the randtreat command, where misfits are assigned in equal proportions to treatment and control. All analyses will control for fixed effects in the variables that the randomization stratified on: daycare center, above/below median language development within age group at baseline, and age-group at endline.
Randomization unit
Randomization of treatment status was done at the family level, which means that all parents to one child receive the same treatment. To have the cleanest treatment possible, children were deemed to be in the same family if they shared a parental phone number, which means that stepsiblings/stepparents were also deemed to belong to the same family. To ensure that all children within a family were assigned the same treatment status, older siblings were removed from the dataset before randomizing.
Clusters
3,181 families
Observations
3,498 children
Arms
1,591 families (1,752 children) in the treatment group and 1,590 families (1,746 children) in the control group.
Primary outcomes
The primary outcome of this study will be children’s language skills as measured by the Danish language assessment tool, “Sprogvurdering 3-6.” The language assessment tool is designed to measure the language skills of children in preschool (ages 3 to 6) and in the first year of primary school (ages 5 or 6). The tool was developed as part of a federal requirement that all public schools perform language assessment on all children the year they start primary school (kindergarten class), and all public preschools perform language assessment on all three-year-old children who are suspected of lagging behind in language development. In practice, many preschools use this test more widely than required as a tool to identify the children who lag behind in their language development and therefore need extra language stimulation. For example, in the year before the intervention, the collaborating preschools assessed the language of 81% of 3-year-olds, 12% of 4-year-olds, and 53% of 5-year-old children, on average, using the tool “Sprogvurdering 3-6.” As part of the RCT, the partnering preschools will increase the scope of their assessments for three consecutive years to include all preschool children included in the trial as well as change the timing of the assessments to fit with the timing of the trial. Specifically, the first round of endline data collection will be initiated immediately after the end of the intervention in the fall of 2020. The second and third rounds of endline data collection will be initiated in the fall of 2021 and 2021, respectively. We plan to conduct the following analyses: 1. Causal effect of the intervention on language skills in the short run: In a linear regression framework, we will compare the language skills measured immediately after the end of the intervention of children in the treatment group to those in the control group. Because the main outcome is a composite measure, we will also conduct reverse regression using machine learning to identify whether there is any function of the outcomes that predicts treatment assignment (Ludwig, Mullainathan, and Spiess 2019). 2. Subgroup analysis of the effect of the intervention on language skills in the short run: We hypothesize that the intervention benefits three groups in particular: children who score below the median in the baseline language assessment, children of parents without a university degree, and children of immigrants. We will explore heterogeneous treatment effects across the distribution of baseline language skills and parental education levels. Our sample does not include enough children of immigrants to provide power to detect a plausible effect size difference for children of immigrants, but the analysis will include explorative subgroup analyses. Furthermore, using a machine-learning framework, we will explore heterogeneous treatment effects in children’s language skills in relation to a variety of demographic characteristics, such as parental education, income, and whether the parents are immigrants, to allow these characteristics to interact with each other in a way that the linear model cannot capture. 3. Causal effect of the intervention on language skills in the medium run: We will repeat exercises 1. and 2. for the endline data collected one and two years, respectively, after the end of the intervention.
Secondary outcomes
As the primary outcome measure is a composite measure of the different subtests in the language assessment “Sprogvurdering 3-6,” the individual subtests will be considered secondary outcomes. These outcomes will be the percentage correctly answered questions in the subtests language comprehension, receptive vocabulary, rhyming, written language awareness, letter recognition, word detection, sound discrimination, as well as the total score in communication strategies. In addition, as the composite measure treats missing answers as incorrect, the total number of missing answers across subtests will be considered a secondary measure. As missing answers are typically a result of the child’s inattention (either because the individual question is too difficult or because the test is abandoned due to lack of focus) this measure is a proxy for the child’s engagement in the language assessment. We plan to conduct the following analyses: 1. Causal effect of the intervention on each subtest: In a linear regression framework, we will compare the percentage correct answers in each subtest in the language assessment for children in the treatment group to those in the control group, while controlling for multiple hypothesis testing. This test will allow us to evaluate whether the intervention affected the different dimensions of language development differently. 2. Causal effect of the intervention on the proportion of answered questions Similarly, in a linear regression framework, we will compare the percentage of missing answers across all subtests in the language assessment for children in the treatment group to those in the control group. In case we find no main effect on the composite measure of language development, this test will allow us to evaluate whether the intervention has affected the children’s engagement in the language assessment.

Cited by

No citation data available for this registration yet.

Keywords