Seeing is Believing? A Cognitive View of Program Take-up
How to cite
Tebes, Jonathan. 2024. "Seeing is Believing? A Cognitive View of Program Take-up." AEA RCT Registry. AEARCTR-0013660. https://doi.org/10.1257/rct.13660-2.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
The Excel Centers (TEC) are tuition-free charter public high schools that award high school diplomas to adults and provide child care, housing and transportation assistance, college credit, and industry-recognized certification courses to help students overcome barriers and continue their education. Through a randomized controlled trial, we will investigate the impact of two behavioral interventions on enrollment and persistence in TEC, with a possible follow-up study examining the impact of increased enrollment and graduation on long-term educational, financial, and labor market outcomes.
The first intervention involves a one-on-one orientation with a life coach before the start of classes. During these orientations, the coach first leads the applicant through a mindfulness breathing exercise before engaging in mental imagery. This involves visualizing their graduation and dream job, as well as writing a letter to their future selves on the day of their graduation. Such exercises have proven effective in other contexts, especially among populations with a history of trauma (Ashraf et al., 2022). The goal of this intervention is to alter “what comes to mind” for students (Gennaioli and Shleifer, 2010): where students previously might think of past difficulties in school, now they may think of their coach and their visualized future (e.g. career or educational goals). By also facilitating a close bond with coaches, this intervention could cascade to increased student engagement and persistence in the program. The second intervention is an advertising video that seeks to reduce perceived barriers of program attendance: depending on the recipient’s age, it features either a single mother or middle-aged man navigating barriers to success at TEC to eventually earn their diplomas.
The research team will enroll 5,000 individuals into the study over 18 months, with enrollees being randomly assigned into one of four groups: (i) being offered only the video, (ii) being offered only the one-on-one orientation, (iii) being offered both the video and the orientation, and (iv) being offered only TEC’s standard services, which do not include either the video or the one-on-one orientation.
Amendment 12/10/2025:
After observing preliminary results from the video intervention, the research team, in collaboration with TEC, has decided to randomize a new set of informational messages. Once their applications are approved, consenting applicants to TEC will be randomized into one of four messaging groups: the first group is a pure control group, which will not receive any messages; the second will receive a standard welcome message; the third group will receive a message that draws on Brough et al. (2024) to provide information on the average earnings effects of the program five years after application; the fourth group will receive information about additional support services available to enrolled students, such as transportation assistance and childcare. 40% of the prospective sample will be assigned to the pure control group and 20% of the sample will be assigned to each of the other three messaging interventions. These messaging interventions will be implemented for roughly 3-9 months to build up a sufficient sample size to evaluate their effectiveness. Then, the research team will present information on orientation participation rates to inform the refinement of future messages and/or the development of new messages.
Citations:
Ashraf, N., Bryan, G., Delfino, A., Holmes, E., Iacovone, L., & Pople, A. (2022). Learning to see the world’s opportunities: The impact of mental imagery on entrepreneurial action. (Working Paper). Innovations for Poverty Action. https://poverty-action.org/sites/default/files/2023-07/learning-to-see-a-world-of-opportunities-working-paper-1.pdf
Gennaioli, Nicola, and Andrei Shleifer. "What comes to mind." The Quarterly journal of economics 125, no. 4 (2010): 1399-1433.
Intervention
Historically, following registration, TEC invited participants to a group orientation. Given the group setting, these orientations allowed little flexibility in scheduling and few opportunities for participants’ individual concerns to be addressed. As part of the study, we will randomize students to attend either the traditional group orientation or a one-on-one orientation with their life coach. Life coaches are dedicated Excel Center staff that students can turn to should they need counseling or resources to address both academic and non-academic barriers. Importantly, coaches often live in the communities they serve and closely reflect the racial and socio-economic backgrounds of their students. During these orientations, students and their life coaches will review the student’s high school records, availability for classes, employment status, and childcare needs. Through a barriers assessment, the life coach works with the student to create a plan that addresses any factors that could impede their ability to attend classes, such as housing instability, legal/justice issues, mental health concerns, or transportation issues. Small amounts of financial assistance are available to address these barriers.
After the barriers assessment, applicants are guided through a breathing-based mindfulness exercise that encourages them to “slow down” and meaningfully engage with the two behavioral exercises that follow (Khaneman, 2011). The first behavioral exercise is a mental imagery exercise where applicants visualize their graduation from TEC and starting their dream job. The second exercise builds on practices that TEC has previously found effective: students write a letter to their future self to open on the day of their graduation. Finally, coaches end the orientation with a positive affirmation that acknowledges the student’s courage to make this commitment to their education and expresses the coach’s belief in them. This intervention builds on promising behavioral interventions in psychology and behavioral economics. Exercises like visualization, mindfulness, and writing a letter to one’s future self have been found to impact education-related and economic outcomes in a wide range of settings. In our context, past negative experiences in school may similarly generate psychological barriers that reduce engagement in adult educational programs. While extensive prior work in psychology has documented that positive mental imagery and letter-to-future-self exercises are associated with improved goal-setting and academic effort (Oyserman et al., 2006; Blouin-Hudon et al., 2017; Yuta et al., 2021), these interventions have yet to be rigorously evaluated as the means for improving enrollment and persistence in educational services.
Soon after application, applicants are also randomized into an intervention group that either does or does not receive an email and a text message that includes one of two “de-stigmatizing” videos. The first video, which is sent if an applicant is below the age of 25, accompanies a young mother as she attends an Excel Center, overcoming her doubts about her place in the program and eventually earning her diploma. The second video, which is sent to those over 25, features a middle-aged man who overcomes his stigma about age to attend classes, graduate, and attain his dream job. These videos were designed by TEC’s marketing team and reflect the two most common student profiles.
Amendment 12/10/2025: Informational Interventions
Preliminary analyses of the data from the video intervention indicate that study applicants are sensitive to the tone and content of information provided by TEC after their application. The video intervention was found to reduce orientation attendance and course enrollment. Since the videos described how past participants were able to overcome significant personal barriers to graduate TEC, they may have unintentionally primed applicants to consider the barriers they face and discouraged them from enrolling in TEC.
Contingent on consent to participate in research, applicants to TEC will now be randomized into one of three text message streams with 20% probability each: research-based, logistics-based, and placebo. The remaining 40% of applications will be randomized into the control group, which will not receive the above messages.
The research team will monitor the effect of each messaging intervention on short-term outcomes at TEC, such as orientation attendance, enrollment, course completion, and graduation. As the enrollment numbers for these interventions increase and their impact is determined, the research team will remove the less effective messages, increasing the treatment probability of the remaining messages, or introducing new messages that build on the findings. All new messages and changes in the randomization probability will be integrated into this registry entry. The text of the current messages is given below:
(20%) T1 (Research): “Welcome to the Excel Center community! Did you know that the average Excel Center graduate sees their earnings increase by 40% in just five years-and many see even bigger gains? We'll be in touch shortly to schedule your orientation.”
(20%) T2 (Logistics): “Welcome to the Excel Center community! Did you know that the Excel Center provides childcare, transportation and other assistance for orientation and beyond? We will be in touch shortly to schedule your orientation.”
(20%) Control (Placebo): “Welcome to the Excel Center community! We're excited to help you begin your journey toward earning your diploma. We will be in touch shortly to schedule your orientation and share more details about what to expect.”
(40%) Pure control: no message
Citations:
Blouin-Hudon, E. M. C. & Pychyl, T. A. (2017). A Mental Imagery Intervention to Increase Future Self-Continuity and Reduce Procrastination. Applied Psychology: An International Review, 66(2), 326–352. https://iaap-journals.onlinelibrary.wiley.com/doi/epdf/10.1111/apps.12088
Khaneman, Daniel. Thinking, fast and slow. Macmillan, 2011.
Oyserman, D., Bybee, D., & Terry, K. (2006). Possible selves and academic outcomes: How and when possible selves impel action. Journal of Personality and Social Psychology, 91(1), 188–204. https://doi.org/10.1037/0022-3514.91.1.188
Yuta, C. & Wilson, A. (2021). Conversation with a future self: A letter-exchange exercise enhances student self-continuity, career planning, and academic thinking. Self and Identity, 20(5), 646-671. https://www.tandfonline.com/doi/full/10.1080/15298868.2020.1754283
Experimental design
In order to be eligible for the study, applicants must be an Indiana resident, not have previously earned their high school diploma, not be a sex offender, and be at least 18 years of age. With the exception of the age restriction, these eligibility criteria exactly follow those of TEC.
Early evidence from a pilot study of these interventions indicates that English language learners may react differently to the behavioral interventions than their peers: life coaches have stated that this group is largely seeking English classes, rather than a high school diploma. As such, it may be the case that the intervention is not a good fit for these students, since they are not facing the same barriers that the intervention seeks to address. Therefore, the research team plans to (1) study English language learners as a sub-population, and (2) based on data from the initial pilot decide with the GW team whether to include this population in the roll-out of the intervention to all ten schools.
The study process begins with potential students filling out an online application form. As part of this form, they are presented with a shortened consent form and given the option of opting out of research participation. Eligible applicants are then randomized with ¼ probability into four groups: a pure control group that receives standard services (i.e., a group orientation and no video message), a group that receives only the video intervention, a group that receives only the one-on-one orientation, and a group that receives both the video and the one-on-one orientation interventions . Additionally, we will randomize all past and existing students of TEC to one of these four groups at the start of the study; this randomization will be stratified based on whether or not their primary language is English and if they have previously enrolled and not just applied for a term at TEC. If these students return to TEC after time away, they will be offered the take-up interventions based on this initial randomization.
We expect nearly full compliance for the video intervention, as it will be texted to the cell phone provided in the application. However, there may not be full compliance for the one-on-one orientation: some individuals may repeatedly miss their scheduled one-on-one orientation dates, in which case they are reverted to the group orientation.
For the analysis, our primary specification estimates the impact of the offer of each intervention, or intention-to-treat (ITT) effects, on outcomes. Since the offer is randomly assigned, any differences in outcomes between groups can be attributed to the intervention offer. The basic specification is (we encase subscripts in parentheses for clarity):
y(ist)= β(1)*Video(it) + β(2)*Orientation(it) + τ(st) + γX(it) + ε(ist)
Where y(ist) is one of our primary take-up outcomes (e.g., TEC enrollment or graduation) for applicant i in application term t, where s denotes the applicant’s randomization strata. Video(it) indicates whether or not participant i was sent the video intervention by term t. Similarly, Orientation(it) indicates whether participant i had been assigned to the one-on-one orientation by term t. τ(st) are strata-time fixed effects, which will include a set of application term indicators and two indicators used in the stratified randomization of prior applicants. Lastly, X(it) is a vector of control variables, such as demographics (age, race, gender) and pre-study variables captured in the application form or historical program data. The coefficients of interest – β(1) and β(2) – respectively estimate the average difference in outcomes between each intervention group (video or one-on-one orientation) and the control group, controlling for baseline characteristics.
Using historical (out-of-sample) data, we will predict a given applicant’s likelihood of enrolling, persisting, and graduating from the program. We anticipate that students who enroll in remedial classes and students who are first-time applicants will be less likely to enroll, persist, and graduate from TEC. We will explore heterogeneity by predicted persistence as well as by key predictors of persistence (e.g. enrolled in remedial classes, English Language Learners, etc.) to understand how the program interacts with the barriers applicants face at baseline. Does the program nudge those on the margin of persisting at baseline? And/or does it facilitate persistence among those least likely to persist at baseline?
Given that our main outcomes relate to take-up, we will also report differences in average characteristics of those who enrolled, persisted or graduated across Treatment and Control groups using a LATE framework (Angrist and Imbens, 1996). This approach instruments for program enrollment, persistence (e.g. any receipt of credit), or program graduation with treatment status, and the dependent variable is a characteristic. Characteristics may include race, age, gender, previous applicant status, predicted academic level upon entry, or English Language Learner status. We will pre-specify the characteristics to be included based on pilot data and will adjust for multiple hypothesis testing using randomization inference. This will provide rich information on whether the interventions differentially benefit those who face the most significant barriers.
We will also explore heterogeneity by remedial class status and predicted program completion. GCSC has indicated that students who enroll in remedial courses, particularly for English and math, are at a higher risk of exiting from TEC due to the longer time to completion for diplomas. Therefore, strong treatment effects from the one-on-one intervention may be indicative of its ability to reframe the importance of a high school diploma for this population. Additionally, if English Language Learners are included in the main analysis, we will explore effects for this sub-sample of students, as they face unique barriers relative to the full sample.
Finally, as part of our analysis, we would like to explore treatment effect heterogeneity using machine learning methods, such as LASSO (Chernozhukov et al., 2018; Davis & Heller, 2017). These will allow us to ascertain which sub-populations in the study were most affected by the treatment. The benefit of this approach is that it does not require the researchers to know a priori which characteristics or interactions of characteristics are related to underlying heterogeneity; the algorithm identifies these characteristics while penalizing over-fitting.
Citations:
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association 91(434), 444-455. https://www.jstor.org/stable/2291629
Chernozhukov, V., Demirer, M., Duflo, E., & Fernandez-Val, I. (2018). Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in India. (Working Paper No. w24678). National Bureau of Economic Research. https://www.nber.org/system/files/working_papers/w24678/w24678.pdf
Davis, J. & Heller, S. B. (2017). Using causal forests to predict treatment heterogeneity: An application to summer jobs. American Economic Review 107(5), 546–50. https://www.jstor.org/stable/44250458
Design details
Cited by
No citation data available for this registration yet.