Seeing is Believing? A Cognitive View of Program Take-up

How to cite

Tebes, Jonathan. 2024. "Seeing is Believing? A Cognitive View of Program Take-up." AEA RCT Registry. AEARCTR-0013660. https://doi.org/10.1257/rct.13660-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jonathan Tebes*

* Primary investigator

Registry ID
AEARCTR-0013660
DOI
https://doi.org/10.1257/rct.13660-2.0
First registered
Last updated
Status
On going
Trial dates
2024-03-18 – 2027-03-22
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/13660

Abstract

The Excel Centers (TEC) are tuition-free charter public high schools that award high school diplomas to adults and provide child care, housing and transportation assistance, college credit, and industry-recognized certification courses to help students overcome barriers and continue their education. Through a randomized controlled trial, we will investigate the impact of two behavioral interventions on enrollment and persistence in TEC, with a possible follow-up study examining the impact of increased enrollment and graduation on long-term educational, financial, and labor market outcomes.

The first intervention involves a one-on-one orientation with a life coach before the start of classes. During these orientations, the coach first leads the applicant through a mindfulness breathing exercise before engaging in mental imagery. This involves visualizing their graduation and dream job, as well as writing a letter to their future selves on the day of their graduation. Such exercises have proven effective in other contexts, especially among populations with a history of trauma (Ashraf et al., 2022). The goal of this intervention is to alter “what comes to mind” for students (Gennaioli and Shleifer, 2010): where students previously might think of past difficulties in school, now they may think of their coach and their visualized future (e.g. career or educational goals). By also facilitating a close bond with coaches, this intervention could cascade to increased student engagement and persistence in the program. The second intervention is an advertising video that seeks to reduce perceived barriers of program attendance: depending on the recipient’s age, it features either a single mother or middle-aged man navigating barriers to success at TEC to eventually earn their diplomas.

The research team will enroll 5,000 individuals into the study over 18 months, with enrollees being randomly assigned into one of four groups: (i) being offered only the video, (ii) being offered only the one-on-one orientation, (iii) being offered both the video and the orientation, and (iv) being offered only TEC’s standard services, which do not include either the video or the one-on-one orientation.

Amendment 12/10/2025:
After observing preliminary results from the video intervention, the research team, in collaboration with TEC, has decided to randomize a new set of informational messages. Once their applications are approved, consenting applicants to TEC will be randomized into one of four messaging groups: the first group is a pure control group, which will not receive any messages; the second will receive a standard welcome message; the third group will receive a message that draws on Brough et al. (2024) to provide information on the average earnings effects of the program five years after application; the fourth group will receive information about additional support services available to enrolled students, such as transportation assistance and childcare. 40% of the prospective sample will be assigned to the pure control group and 20% of the sample will be assigned to each of the other three messaging interventions. These messaging interventions will be implemented for roughly 3-9 months to build up a sufficient sample size to evaluate their effectiveness. Then, the research team will present information on orientation participation rates to inform the refinement of future messages and/or the development of new messages.

Citations:

Ashraf, N., Bryan, G., Delfino, A., Holmes, E., Iacovone, L., & Pople, A. (2022). Learning to see the world’s opportunities: The impact of mental imagery on entrepreneurial action. (Working Paper). Innovations for Poverty Action. https://poverty-action.org/sites/default/files/2023-07/learning-to-see-a-world-of-opportunities-working-paper-1.pdf

Gennaioli, Nicola, and Andrei Shleifer. "What comes to mind." The Quarterly journal of economics 125, no. 4 (2010): 1399-1433.

Intervention

Historically, following registration, TEC invited participants to a group orientation. Given the group setting, these orientations allowed little flexibility in scheduling and few opportunities for participants’ individual concerns to be addressed. As part of the study, we will randomize students to attend either the traditional group orientation or a one-on-one orientation with their life coach. Life coaches are dedicated Excel Center staff that students can turn to should they need counseling or resources to address both academic and non-academic barriers. Importantly, coaches often live in the communities they serve and closely reflect the racial and socio-economic backgrounds of their students. During these orientations, students and their life coaches will review the student’s high school records, availability for classes, employment status, and childcare needs. Through a barriers assessment, the life coach works with the student to create a plan that addresses any factors that could impede their ability to attend classes, such as housing instability, legal/justice issues, mental health concerns, or transportation issues. Small amounts of financial assistance are available to address these barriers.

After the barriers assessment, applicants are guided through a breathing-based mindfulness exercise that encourages them to “slow down” and meaningfully engage with the two behavioral exercises that follow (Khaneman, 2011). The first behavioral exercise is a mental imagery exercise where applicants visualize their graduation from TEC and starting their dream job. The second exercise builds on practices that TEC has previously found effective: students write a letter to their future self to open on the day of their graduation. Finally, coaches end the orientation with a positive affirmation that acknowledges the student’s courage to make this commitment to their education and expresses the coach’s belief in them. This intervention builds on promising behavioral interventions in psychology and behavioral economics. Exercises like visualization, mindfulness, and writing a letter to one’s future self have been found to impact education-related and economic outcomes in a wide range of settings. In our context, past negative experiences in school may similarly generate psychological barriers that reduce engagement in adult educational programs. While extensive prior work in psychology has documented that positive mental imagery and letter-to-future-self exercises are associated with improved goal-setting and academic effort (Oyserman et al., 2006; Blouin-Hudon et al., 2017; Yuta et al., 2021), these interventions have yet to be rigorously evaluated as the means for improving enrollment and persistence in educational services.

Soon after application, applicants are also randomized into an intervention group that either does or does not receive an email and a text message that includes one of two “de-stigmatizing” videos. The first video, which is sent if an applicant is below the age of 25, accompanies a young mother as she attends an Excel Center, overcoming her doubts about her place in the program and eventually earning her diploma. The second video, which is sent to those over 25, features a middle-aged man who overcomes his stigma about age to attend classes, graduate, and attain his dream job. These videos were designed by TEC’s marketing team and reflect the two most common student profiles.

Amendment 12/10/2025: Informational Interventions

Preliminary analyses of the data from the video intervention indicate that study applicants are sensitive to the tone and content of information provided by TEC after their application. The video intervention was found to reduce orientation attendance and course enrollment. Since the videos described how past participants were able to overcome significant personal barriers to graduate TEC, they may have unintentionally primed applicants to consider the barriers they face and discouraged them from enrolling in TEC.

Contingent on consent to participate in research, applicants to TEC will now be randomized into one of three text message streams with 20% probability each: research-based, logistics-based, and placebo. The remaining 40% of applications will be randomized into the control group, which will not receive the above messages.

The research team will monitor the effect of each messaging intervention on short-term outcomes at TEC, such as orientation attendance, enrollment, course completion, and graduation. As the enrollment numbers for these interventions increase and their impact is determined, the research team will remove the less effective messages, increasing the treatment probability of the remaining messages, or introducing new messages that build on the findings. All new messages and changes in the randomization probability will be integrated into this registry entry. The text of the current messages is given below:

(20%) T1 (Research): “Welcome to the Excel Center community! Did you know that the average Excel Center graduate sees their earnings increase by 40% in just five years-and many see even bigger gains? We'll be in touch shortly to schedule your orientation.”

(20%) T2 (Logistics): “Welcome to the Excel Center community! Did you know that the Excel Center provides childcare, transportation and other assistance for orientation and beyond? We will be in touch shortly to schedule your orientation.”

(20%) Control (Placebo): “Welcome to the Excel Center community! We're excited to help you begin your journey toward earning your diploma. We will be in touch shortly to schedule your orientation and share more details about what to expect.”

(40%) Pure control: no message

Citations:

Blouin-Hudon, E. M. C. & Pychyl, T. A. (2017). A Mental Imagery Intervention to Increase Future Self-Continuity and Reduce Procrastination. Applied Psychology: An International Review, 66(2), 326–352. https://iaap-journals.onlinelibrary.wiley.com/doi/epdf/10.1111/apps.12088
Khaneman, Daniel. Thinking, fast and slow. Macmillan, 2011.

Oyserman, D., Bybee, D., & Terry, K. (2006). Possible selves and academic outcomes: How and when possible selves impel action. Journal of Personality and Social Psychology, 91(1), 188–204. https://doi.org/10.1037/0022-3514.91.1.188

Yuta, C. & Wilson, A. (2021). Conversation with a future self: A letter-exchange exercise enhances student self-continuity, career planning, and academic thinking. Self and Identity, 20(5), 646-671. https://www.tandfonline.com/doi/full/10.1080/15298868.2020.1754283

Experimental design

In order to be eligible for the study, applicants must be an Indiana resident, not have previously earned their high school diploma, not be a sex offender, and be at least 18 years of age. With the exception of the age restriction, these eligibility criteria exactly follow those of TEC.

Early evidence from a pilot study of these interventions indicates that English language learners may react differently to the behavioral interventions than their peers: life coaches have stated that this group is largely seeking English classes, rather than a high school diploma. As such, it may be the case that the intervention is not a good fit for these students, since they are not facing the same barriers that the intervention seeks to address. Therefore, the research team plans to (1) study English language learners as a sub-population, and (2) based on data from the initial pilot decide with the GW team whether to include this population in the roll-out of the intervention to all ten schools.

The study process begins with potential students filling out an online application form. As part of this form, they are presented with a shortened consent form and given the option of opting out of research participation. Eligible applicants are then randomized with ¼ probability into four groups: a pure control group that receives standard services (i.e., a group orientation and no video message), a group that receives only the video intervention, a group that receives only the one-on-one orientation, and a group that receives both the video and the one-on-one orientation interventions . Additionally, we will randomize all past and existing students of TEC to one of these four groups at the start of the study; this randomization will be stratified based on whether or not their primary language is English and if they have previously enrolled and not just applied for a term at TEC. If these students return to TEC after time away, they will be offered the take-up interventions based on this initial randomization.

We expect nearly full compliance for the video intervention, as it will be texted to the cell phone provided in the application. However, there may not be full compliance for the one-on-one orientation: some individuals may repeatedly miss their scheduled one-on-one orientation dates, in which case they are reverted to the group orientation.

For the analysis, our primary specification estimates the impact of the offer of each intervention, or intention-to-treat (ITT) effects, on outcomes. Since the offer is randomly assigned, any differences in outcomes between groups can be attributed to the intervention offer. The basic specification is (we encase subscripts in parentheses for clarity):

y(ist)= β(1)*Video(it) + β(2)*Orientation(it) + τ(st) + γX(it) + ε(ist)

Where y(ist) is one of our primary take-up outcomes (e.g., TEC enrollment or graduation) for applicant i in application term t, where s denotes the applicant’s randomization strata. Video(it) indicates whether or not participant i was sent the video intervention by term t. Similarly, Orientation(it) indicates whether participant i had been assigned to the one-on-one orientation by term t. τ(st) are strata-time fixed effects, which will include a set of application term indicators and two indicators used in the stratified randomization of prior applicants. Lastly, X(it) is a vector of control variables, such as demographics (age, race, gender) and pre-study variables captured in the application form or historical program data. The coefficients of interest – β(1) and β(2) – respectively estimate the average difference in outcomes between each intervention group (video or one-on-one orientation) and the control group, controlling for baseline characteristics.

Using historical (out-of-sample) data, we will predict a given applicant’s likelihood of enrolling, persisting, and graduating from the program. We anticipate that students who enroll in remedial classes and students who are first-time applicants will be less likely to enroll, persist, and graduate from TEC. We will explore heterogeneity by predicted persistence as well as by key predictors of persistence (e.g. enrolled in remedial classes, English Language Learners, etc.) to understand how the program interacts with the barriers applicants face at baseline. Does the program nudge those on the margin of persisting at baseline? And/or does it facilitate persistence among those least likely to persist at baseline?

Given that our main outcomes relate to take-up, we will also report differences in average characteristics of those who enrolled, persisted or graduated across Treatment and Control groups using a LATE framework (Angrist and Imbens, 1996). This approach instruments for program enrollment, persistence (e.g. any receipt of credit), or program graduation with treatment status, and the dependent variable is a characteristic. Characteristics may include race, age, gender, previous applicant status, predicted academic level upon entry, or English Language Learner status. We will pre-specify the characteristics to be included based on pilot data and will adjust for multiple hypothesis testing using randomization inference. This will provide rich information on whether the interventions differentially benefit those who face the most significant barriers.

We will also explore heterogeneity by remedial class status and predicted program completion. GCSC has indicated that students who enroll in remedial courses, particularly for English and math, are at a higher risk of exiting from TEC due to the longer time to completion for diplomas. Therefore, strong treatment effects from the one-on-one intervention may be indicative of its ability to reframe the importance of a high school diploma for this population. Additionally, if English Language Learners are included in the main analysis, we will explore effects for this sub-sample of students, as they face unique barriers relative to the full sample.

Finally, as part of our analysis, we would like to explore treatment effect heterogeneity using machine learning methods, such as LASSO (Chernozhukov et al., 2018; Davis & Heller, 2017). These will allow us to ascertain which sub-populations in the study were most affected by the treatment. The benefit of this approach is that it does not require the researchers to know a priori which characteristics or interactions of characteristics are related to underlying heterogeneity; the algorithm identifies these characteristics while penalizing over-fitting.

Citations:

Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association 91(434), 444-455. https://www.jstor.org/stable/2291629

Chernozhukov, V., Demirer, M., Duflo, E., & Fernandez-Val, I. (2018). Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in India. (Working Paper No. w24678). National Bureau of Economic Research. https://www.nber.org/system/files/working_papers/w24678/w24678.pdf

Davis, J. & Heller, S. B. (2017). Using causal forests to predict treatment heterogeneity: An application to summer jobs. American Economic Review 107(5), 546–50. https://www.jstor.org/stable/44250458

Design details

Randomization method
New TEC applicants will be randomly assigned into a treatment group using random number generation in Goodwill Indy’s student information system. Previous applicants who are returning to TEC will be randomized via STATA using a stratified design: the strata are based on whether the applicant’s primary language is English and whether the applicant had previously completed a term at TEC.
Randomization unit
Individual
Clusters
N/A: the treatment is not clustered
Observations
5,000 applicants
Arms
1,250 receive only the video 1,250 receive only the one-on-one orientation 1,250 receive both the video and the one-on-one orientation 1,250 receive standard services (neither the video nor the one-on-one orientation) Amendment 12/10/2025: We will enroll TEC applicants into the text message streams for roughly 3-9 months to build up a sufficient sample size to evaluate their effectiveness. Then, the research team will present information on orientation participation rates to inform the refinement of future messages and/or the development of new messages.
Primary outcomes
Our primary outcomes at the applicant-level will be (individual outcomes listed on each line): Indicator for attending an orientation within one year after application Indicator for being enrolled in a class at TEC within one year after application Indicator for earning any credit from the TEC within one year after application Indicator for the receipt of a high school diploma from TEC within one year after application
Secondary outcomes
Our secondary take-up outcomes at the applicant-level will be (first line of each grouping is the category, followed by specific outcomes): Student interactions with life coaches (grouping): Cumulative count of separate interactions with life coach (including orientation) Cumulative count of separate calls or text conversations Cumulative count of in-person meetings Enrollment at TEC, such as (grouping): Cumulative count of terms enrolled in classes at TEC Cumulative count of terms with any credits earned Class attendance at TEC, such as (grouping): Indicator for first-day attendance in at least one of the applicant’s enrolled courses in the term of application Proportion of class days marked as attended out of total number of enrolled class days Cumulative count of total days marked as attended across all terms Metrics for academic performance at TEC, such as (grouping): Grade point average Cumulative count of credits earned Cumulative count of math credits earned at TEC Cumulative count of English credits earned at TEC Indicator for received any credit towards post-secondary degree or certificate Usage of non-academic services from TEC, such as (grouping): Indicator for usage of any non-academic services from TEC Indicator for usage of TEC’s housing services Indicator for usage of TEC’s mental health services Indicator for usage of TEC’s childcare services Indicator for usage of TEC’s transportation services Indicator for usage of other TEC services Estimated cost of all TEC services used As a potential follow-up study, we would also like to investigate the effects of the behavioral interventions on a set of long-term outcomes. Depending on data availability, these outcomes could include (structured as above, but with groupings and sub-groupings): Labor market outcomes (grouping): Total quarterly earnings Indicator for employment Count of quarters employed Length of applicant’s longest spell of employment Education and professional training outcomes (grouping): Progress towards and receipt of a professional certificate (sub-grouping): Indicator for receipt of a professional certificate in a given field Count of credits towards a professional certificate Post-secondary institution attendance and degree receipt (sub-grouping): Indicator for receiving a degree in a given field Indicator for receipt of credits toward a degree in a given field Count of credits towards a degree in a given field Indicator for attendance at a 2 year institution Indicator for attendance at a 4 year institution Indicator for graduation from a 2 year institution Indicator for graduation from a 4 year institution Utilization of Public Benefits (grouping): Indicator for receiving public housing support (housing vouchers, rental assistance, etc.) Expenditures received related to housing (housing vouchers, rental assistance, etc.) Indicator for enrollment in Supplemental Nutritional Assistance Program (SNAP) Expenditures received from SNAP Indicator for enrollment in Temporary Assistance for Needy Families (TANF) Expenditures received from TANF Credit and delinquency outcomes (grouping): Credit Index (sub-grouping): Credit score estimated using VantageScore Total available credit on all accounts Total balance on all accounts Delinquency Index (sub-grouping): Amount past due on trades presently 30 days delinquent reported in the last 6 months Amount of debt past due held by third-party collection agencies Total number of public record bankruptcies and tax liens Total number of trades presently satisfactory that were ever 30 or more days delinquent or derogatory excluding collections Criminal justice involvement (grouping): Indicator for any criminal charge Indicator for any felony charge Indicator for any misdemeanor charge Indicator for any violation charge Cumulative count of criminal charges Cumulative count of felony charges Cumulative count of misdemeanor charges Cumulative count of violation charges

Cited by

No citation data available for this registration yet.

Keywords