Improving survey completions using additional reminder emails and by providing additional information on survey content and impact

How to cite

Meer, Janna Ter. 2022. "Improving survey completions using additional reminder emails and by providing additional information on survey content and impact." AEA RCT Registry. AEARCTR-0008792. https://doi.org/10.1257/rct.8792-2.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Janna Ter Meer*

* Primary investigator

Registry ID
AEARCTR-0008792
DOI
https://doi.org/10.1257/rct.8792-2.1
First registered
Last updated
Status
Completed
Trial dates
2022-01-12 – 2023-12-30
Countries
United States of America (California)
Registry record
http://www.socialscienceregistry.org/trials/8792

Abstract

This study looks at:
* Determining whether a ‘pure’ reminder is more effective at increasing survey completion in a health research program than a reminder that also communicates the content of the survey and its impact.
* Determining if there are any spillover effects of being in a given reminder group (business-as-usual 2 reminders; larger set of pure reminders; larger set of reminders including information on survey impact and content) on subsequent participation.

The All of Us Research Program (AoURP) is an ambitious effort to gather health data from one million or more people living in the United States to accelerate research that may improve health. The aim is to help build one of the most diverse health databases in history. Researchers will use the data to learn how our biology, lifestyle, and environment affect health. This may one day help them find ways to treat and prevent disease. Understanding the health conditions, lifestyle and environmental conditions of participants are key pieces of information for researchers.

PPI 1-3 are important surveys in which AoURP collects data about the demographic background, health history and lifestyle information about our participants. The Basics survey (PPI 1) informs us of key information about the participant, including their age, race, gender, and education level. The Overall Health (PPI 2) survey asks participants about their physical and mental health backgrounds, which enables researchers to understand the health conditions of the AoURP cohort. The Lifestyle (PPI 3) survey asks about the habits of the participant, including smoking, alcohol, and recreational drug use habits. After primary consent completion, roughly 85% of participants complete the Basics (PPI 1) survey (calculated over December 2019 to November 2021). This number drops to 50-51% for PPI 2 and 3, resulting in an overall PPI 1-3 completion of 50%. This experiment aims to determine if additional reminders and variations in the reminder content can encourage more participants to complete the PPI 1-3 surveys.

Intervention

This study will comprise of three arms:
• The control arm which includes two business-as-usual reminder emails.
• The pure reminder arm includes up to 6 reminders and informs the participant that they need to complete a survey. These emails feature a CTA, but don't go into detail about the survey content or its impact.
• The reminder + content + impact arm includes up to 6 reminders that include a CTA at the top as well as information about the survey content and its impact for AoURP researchers.

Update made to registration on 09/22/22:
We are extending this evaluation to two reminders for two additional surveys in the AoU journey called PPI 5 (Health Care Access and Utilization) and PPI 6 (Personal & Family Health History).For each survey, two reminders are sent spaced 5 days apart. The PPI 5 and 6 surveys become available to participants that have completed PPI1 (Basics) and 90 days have passed since program enrollment, which is also when the first reminder for PPI 5 goes out. We will have two conditions: Pure Reminder and Reminder + that follow the same structure as for the PPI 1-3 reminders described above. In both groups, the CTA is in the first paragraph of the email and the email subject line and preview text are the same across groups. In other words, the only difference between the conditions is that the Reminder+ condition gives more detail about the survey content and why the answers are important for health research.

Note that we are choosing not to test the control messages for PPI 5-6 so that we can better power the other two conditions.

Experimental design

The study will take place in the All of Us Research Program (AoURP). This is a longitudinal health research program looking at the impact of genetics and environmental factors on health outcomes, with a particular focus on individuals that are underrepresented in biomedical research. Once the user consents to become a participant, they can share different types of health data with the program, including through surveys, donating a biosample and by linking their Electronic Health Records. Our focus is on participants at the beginning of the journey and their likelihood to complete three important surveys for the program.

AoURP participants are recruited from the general public through advertisements on websites such as webmd.com and nextdoor.com. To date, AoURP has enrolled more than 400,000 participants. We will aim to recruit 21,700 participants. This corresponds to 4,340 participants in the control condition and 8,680 for each of the treatment conditions (17,360 participants for the pooled comparison).

After creating an account on the AoURP platform, users go through the primary consent module. Once the user has given consent to join the program, they become a program participant. Upon completing the primary consent, the participant becomes eligible to complete the Basics survey (PPI 1). This survey informs us of key information about the participant, including their age, race, gender and education level. Once completed, the participant becomes eligible to complete the Overall Health and the Lifestyle surveys. The Overall Health (PPI 2) survey asks participants about their physical and mental health backgrounds, which enables researchers to understand the health conditions of the AoURP cohort. The Lifestyle (PPI 3) survey asks about the habits of the participant, including smoking, alcohol, and recreational drug use habits. Participants are randomized to one of three trial arms after completing primary consent and receive specific reminder messages corresponding to their group assignment if they meet the reminder requirements.

The control includes two business-as-usual reminder emails. The first email is sent 24 hours after completing primary consent AND PPI 1 AND PPI 2 AND PPI 3 are incomplete. The second email is sent 5 days after completing primary consent AND PPI 1 OR PPI 2 OR PPI 3 are incomplete.

The pure reminder arm includes up to 6 pure reminders. The pure reminder emails inform the participant that they need to complete a survey. These emails include a CTA, but don't give detail about the survey content or its impact. In this arm, each PPI survey has two email alternatives for a total of 6 reminder emails. The first PPI 1 email is sent 24 hours after completing primary consent AND PPI 1 is incomplete. The first PPI 2 email is sent 24 hours after completing PPI 1 AND PPI 2 is incomplete. The first PPI 3 email is sent 24 hours after completing PPI 2 AND PPI 3 is incomplete OR 12 days after completing consent AND PPI 1 is complete AND PPI 3 is incomplete. The second PPI 1 email is sent 5 days after completing primary consent AND PPI 1 is incomplete. The second PPI 2 email is sent 5 days after completing PPI 1 AND PPI 2 is incomplete. The second PPI 3 email is sent 5 days after completing PPI 2 AND PPI 3 is incomplete OR after 17 days of completing consent AND PPI 1 is complete AND PPI 3 is incomplete.

The reminder + content + impact arm includes up to 6 reminders that include a CTA at the top as well as information about the survey content and its impact for AoURP researchers. In this arm, each PPI survey has two email alternatives for a total of 6 reminder emails. The first PPI 1 email is sent 24 hours after completing primary consent AND PPI 1 is incomplete. The first PPI 2 email is sent 24 hours after completing PPI 1 AND PPI 2 is incomplete. The first PPI 3 email is sent 24 hours after completing PPI 2 AND PPI 3 is incomplete OR 12 days after completing consent AND PPI 1 is complete AND PPI 3 is incomplete. The second PPI 1 email is sent 5 days after completing primary consent AND PPI 1 is incomplete. The second PPI 2 email is sent 5 days after completing PPI 1 AND PPI 2 is incomplete. The second PPI 3 email is sent 5 days after completing PPI 2 AND PPI 3 is incomplete OR after 17 days of completing consent AND PPI 1 is complete AND PPI 3 is incomplete.

The proportion of participants that complete the respective surveys (PPI 1, 2 and 3) relative to those that complete primary consent after receiving one of three reminder email interventions will be calculated. This measure will assess which intervention was the most successful at improving the conversion rate for the PPI 1-3 surveys.

The study will be active in the AoURP dashboard for an estimated nine weeks allowing for individuals to participate in the study.

Update made to registration on 09/22/22:
See additional documentation for message content and trigger conditions of the PPI 5-6 messages.

Design details

Randomization method
Once participants become eligible for the study, they are randomly assigned a number between 1 and 3, which corresponds to one of the three trial arms. We are overweighting assignment into the treatment. Specifically, for each participant, we generate a random integer ranging from 0 to 4. Individuals being assigned 0 are in the control condition, those assigned 1 or 2 go into the pure reminder condition and those assigned 3 or 4 go into the reminder+ condition. Update made to registration on 09/22/22: Participants are re-randomized into a Pure Reminder or Reminder+ group for the PPI 5-6 reminder series. They will be assigned to either group with equal probability.
Randomization unit
The unit of randomization is at the individual participant level. This happens when participants complete primary consent.
Clusters
N/A
Observations
Approximately 21,700 persons Update made to registration on 09/22/22: Traffic to the platform has slowed since launching the study. We are therefore revising our target sample size to 15,750 persons.
Arms
Approximately 4,340 participants in the control condition and 8,680 for each of the treatment conditions (17,360 participants for the pooled comparison). Update made to registration on 09/22/22: We are seeking to obtain approximately 6,300 in each of the two treatment arms.
Primary outcomes
The primary outcome variable is the proportion of participants that complete the respective surveys (PPI 1, 2 and 3) relative to those that complete primary consent after receiving one of three reminder email interventions. Update made to registration on 09/22/22: For the additional evaluation of the PPI 5-6 reminders, we will look at the proportion of participants that complete PPI 5 and 6 relative to those that complete Primary Consent and PPI 1 (Basics) and receive at least one of the reminder messages. We will analyze results for the pooled surveys (Pure Reminder for PPI 1-3 and PPI 5-6 against Reminder+ for PPI 1-3 and PPI 5-6) and separately by PPI 1-3 and PPI 5-6.

Cited by

No citation data available for this registration yet.

Keywords