Text messaging for parental engagement in student learning

How to cite

Sautmann, Anja. 2020. "Text messaging for parental engagement in student learning." AEA RCT Registry. AEARCTR-0006701. https://doi.org/10.1257/rct.6701-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Anja Sautmann*

* Primary investigator

Registry ID
AEARCTR-0006701
DOI
https://doi.org/10.1257/rct.6701-1.1
First registered
Last updated
Status
Completed
Trial dates
2020-11-02 – 2020-11-30
Countries
Nigeria (Lagos)
Registry record
http://www.socialscienceregistry.org/trials/6701

Abstract

This experiment will use exploration sampling to test the effects of weekly text messaging to parents' phones as a means to increase student use of a phone quiz platform and ultimately learning outcomes.

The study will be implemented by a partner organization that operates private schools in Lagos, Nigeria, among approximately 4705 students in Primaries 1-6. The study will proceed in two stages. In the first stage, we will use adaptive treatment assignment with exploration sampling and stratification to choose one out of four "types" of text messages. Designed by the partner, the four message "types" include motivational framing, testimonial framing, and varying numbers of practice quizzes tailored to what the student recently covered in class. The text message type chosen in the first stage is the one that promises to lead to the greatest increase in use by students of the partner's quiz platform. In the second stage, we will use a standard RCT with a treatment group that receives the message type chosen in the first stage, and a control group that receives no messaging, to test whether quiz use leads to sustained increases in exam scores.

Intervention

Stage 1: The goal is to choose one of four treatments (message type) for implementation in the RCT in Stage 2. In the 5 weeks prior to the winter break 2020 (Nov 2-Dec 07 weeks), we will conduct an experiment with five waves using adaptive treatment assignment. Each treatment consists of a set of text messages to be sent to parents weekly in order to encourage the use of the partner organization's proprietary Whatsapp-based quiz platform. The four treatments consist of four "types" of messages, sent each week to the randomly selected treatment groups:

1. Motivation framing: a set of 10 messages focusing on different motivations for learning.
2. Testimonial framing: a set of 10 messages focusing on the value of the quiz app.
3. One specifi c quiz: proposing 1 quiz tailored to what the student covered in class that
week/term.
4. Many quizzes: proposing several quizzes tailored to what the student covered in class that
week/term.

The outcome is a binary variable indicating "success". It is equal to 1 if at least one quiz was completed within 30 hours from the text message, equal to 0 if no quiz was completed from the phone number contacted.

Stage 2: A treatment group will receive the message type chosen above on a weekly basis for the duration of the school term beginning in January 2021.

Based on the estimated quiz uptake rates in Stage 1, the quiz usage rates prior to the experiment, and grade distributions from prior years, we will calculate the minimal average increase in assignment scores we would need to observe for each child who has adopted
the quizzes, in order to see a significant treatment effect using traditional frequentist inference for treatment and control group. If this increase is unrealistic, the partner organization may decide not to conduct the trial.

Experimental design

We will form three strata: primary 1 and 2, primary 3-5, and primary 6.

First stage: For each phone number, one child associated with this number is selected randomly, generating a sample of 4705 children.

We stratify the sample and randomly select an equal share of each stratum for each of the 5 experimental waves. We then randomly assign children in each stratum and wave to the four treatment arms. In the fi rst wave, assignment to each treatment arm is equal (aside
from "misfi ts" arising from indivisibilities in the total numbers). In all following waves, we use exploration sampling to choose the assignment share to each treatment arm within each stratum.

Second stage: Using the full sample of 6066 children (as of Nov 2, 2020), families will be strati fied by the primary grade level of their child or children, and randomly assigned in equal proportions to the treatment and control group. If for any child in a given family, the most successful treatment type is treatment 3 or 4 (speci fic quizzes), the parent will receive one message for each child in that treatment type. The message will include the name of the child to attribute the correct quizzes to each child. If any children in the same family are assigned messages of type 1 or 2 according to Stage 1 results, the family will receive one message for all those treatment arms. Messages will be randomized according to the proportion of children in each treatment arm (to avoid duplicate messaging with similar contents).

For example, suppose a family has 5 children, 2 in primaries 1 and 2, 2 in primaries 4 and 5, and 1 in primary 6. Primaries 1-2 is assigned treatment arm 3, primaries 3-5 are assigned arm 2, and primary 6 is assigned arm 1. Then this family will receive 3 messages: one for each child in the quiz messaging arm (primaries 1 and 2) and one randomly selected message that is with 2/3 probability treatment arm 2, and 1/3 probability treatment arm 1.

Design details

Randomization method
Assignment according to the predetermined assignment shares (equal shares, or as determined by exploration sampling algorithm) is carried out using the randtreat command in Stata, which carries out factorial assignment within stratum with tie-breaking.
Randomization unit
Parent/phone number
Clusters
4,705 phone numbers
Observations
First stage: 4,705 phone numbers (parents) Second stage: 6,066 children
Arms
First stage: in each wave, in expectation 235 phone number are assigned to each of the four messaging treatment arms. However, assignment shares will vary after the first stage according to exploration sampling assignment shares. Each wave will consist of approximately 941 parents in total receiving messages. Second stage: the sample will be split evenly between treatment and control apartment from any misfits that arise from stratification, i.e., we aim to assign 2,353 families to each treatment and control, with balanced allocation of observations.
Primary outcomes
Stage 1: indicator whether at least 1 practice quiz was taken 30 hours after text message was sent. Stage 2: grade-specific normalized midterm and endterm exam scores.

Cited by

No citation data available for this registration yet.

Keywords