TalkingPoints Evaluation
How to cite
Michelini, Michelle Park; Kalil, Ariel. 2019. "TalkingPoints Evaluation." AEA RCT Registry. AEARCTR-0004814. https://doi.org/10.1257/rct.4814-3.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We evaluate an education technology platform that is designed to increase parent engagement with the goal of improving student engagement in the short run and academic outcomes in the long run. TalkingPoints has two components. TalkingPoints Multilingual Messaging facilitates communication between teachers and parents by providing an easy-to-use platform that promotes two-way translated messages in over 30 languages. TalkingPoints Coach provides parents tips for communicating with teachers and other information about their children’s schooling. Messages between parents and teachers come directly from TalkingPoints. Both local and federal law requires schools to provide programing intended to promote parental engagement in their children’s education. We assess the effectiveness of TalkingPoints in a school-based randomized field experiment over one school year in 50–65 schools using objective measures of parent communication with schools, engagement with children and children’s executive functioning.
Intervention
The Evaluation
The Sample. The evaluation will be a school-based randomized controlled trial that will be implemented in grade three. We focus on one grade to improve power. We selected third grade because it is the point in schooling when academics become more rigorous and consequential in public schools. Developmentally and institutionally, students are expected to have acclimated to the classroom routine and have the executive functioning skills for staying on task and completing homework by third grade, which are elements required for our outcome measures described below. The evaluation will include 50-60 schools in the US. One third-grade classroom in each school will participate in the evaluation. Classes are expected to include about 25 students. There is an oversample of Title I schools and schools with a large English Second Language population.
The Treatment. Students in one third-grade classroom in both treatment and control schools will be enrolled in TalkingPoints for this evaluation. Teachers in treatment classrooms will have access to TalkingPoints while teachers in control classrooms will not. Control group teachers will be subject to “business as usual” in terms of communicating with parents. Prior to randomization, teachers in both treatment and control classrooms will register their students in TalkingPoints by entering demographic into the TalkingPoints registration system. These data will be used to test balance across the treatment and control groups and as covariates in the data analysis. We also plan to collect student attendance data from the school or district.
We will consent teachers and students based on the policies of participating schools. Once students in all participating classrooms are registered and parents have consented, schools will be randomized into either the treatment or the control group. Following randomization, treatment schools will receive TalkingPoints at no cost for the 2020–2021 school year. Control group schools will not receive TalkingPoints in 2020–2021 but will receive it at no cost in the 2021–2022 school year. TalkingPoints will become available to treatment schools in October 2020. The treatment will continue until the end of the school year. All student assessments and parent and teacher surveys will be completed in May 2021.
Experimental design
Communication between Teachers and Parents. As an objective measure of parent-teacher communication, we will use simulated communications to test whether treatment parents are more likely than control parents to communicate with schools. If TalkingPoints increases the chance that a parent will communicate with the school, then when TalkingPoints sends a message to both treatment and control parents asking for a response, parents in the treatment group ought to be more likely than parents in the control group to respond. For example, a simulated message might be, “Did your child have any math homework this week?” The parent would be asked to reply by clicking yes or no. The measure of communication will be the number of replies.
Parent Engagement in Their Child’s Schooling. We cannot observe parent engagement with their child at home and parent reports of their own engagement are likely to be biased, so we will create simulated engagement activities. One of the main ways that schools hope parents will engage their children is by helping them complete their homework on time. To measure parental engagement we will develop simulated homework assignments. Teachers will distribute and collect these simulated assignments as regular classroom assignments. The number of returned homework assignments will be a measure of parent engagement in their child’s schooling, since engaged parents are more likely to monitor their child’s homework.
We will use the simulated assignments to measure the extent to which parents engage directly with their child by having some simulated assignments require parental engagement. We will compare the average number of simulated assignments turned in by control and treatment students. We also will look at differences in the trajectory of the number of simulated assignments turned in over time. We expect that if parents become more engaged with their child’s schooling due to participation in TalkingPoints, the number of assignments turned in will increase for treatment parents but not control parents over time or at least over the first few months, after which the difference may level off.
Child’s Executive Functioning Skill. Executive function is a set of cognitive processes that are required for the control of behavior and include working memory, cognitive flexibility, and inhibitory control. These processes taken as a unitary construct control one’s ability to focus, stay on task when faced with distraction, exercise self control, set and accomplish goals, problem solve, remain flexible during times of transition or change, and manage the daily demands of life. Parent involvement in executive function training and development in children is hypothesized to have a positive impact, and TalkingPoints is a natural experiment to test this theory. We will compare the standardized score differential between pre and post-test periods for treatment and control students.
Parent Survey. We will collect qualitative feedback from parents about TalkingPoints usage and the evaluation at the end of the intervention via text (through TalkingPoints) with incentives.
Teacher Survey. At the end of the intervention we will implement a short survey of both treatment and control teachers to ask about the social, economic, and skill composition of their classrooms. We will also ask treatment teachers about their use of TalkingPoints, although we will have objective data on the fidelity of their use from data collected in TalkingPoints.
Design details
Cited by
No citation data available for this registration yet.