Impact Evaluation of the Boys & Girls Club’s Academic and Emotional Well-Being Enrichment Program

How to cite

Kofoed, Michael; Tebes, Jonathan. 2024. "Impact Evaluation of the Boys & Girls Club’s Academic and Emotional Well-Being Enrichment Program." AEA RCT Registry. AEARCTR-0014404. https://doi.org/10.1257/rct.14404-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Michael Kofoed* , Jonathan Tebes

* Primary investigator

Registry ID
AEARCTR-0014404
DOI
https://doi.org/10.1257/rct.14404-1.1
First registered
Last updated
Status
In development
Trial dates
2024-09-23 – 2030-06-30
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/14404

Abstract

The Wilson Sheehan Lab for Economic Opportunities (LEO) has partnered with the Boys & Girls Club of the Northern Indiana Corridor (BGCNIC), located in South Bend, Indiana, to evaluate their academic and emotional well-being (EWB) enrichment program. Their academic and emotional well-being enrichment program builds upon the usual after-school club experience, combining STEM, literacy, and social-emotional learning components into a single learning program. BGCNIC identified Club members for the program if they are in kindergarten through fifth grade. Historically, the rate of chronic absenteeism for many of the districts served by BGCNIC is well above the state and national averages, and the percent of students testing at or above proficiency in literacy and math is much lower. A key component of this program is the provision of a safe space for young children to learn and develop self-regulatory behaviors that may prove to have an impact on their academic outcomes.
To evaluate this program, we will use a randomized control trial (RCT) to examine whether the program improves grades, test scores, school attendance, and behavioral outcomes. Eligible students are randomly assigned to one of three club tracks: a control group, social-emotional learning (SEL), or SEL plus academic programming, with approximately 600 in each group. The study will utilize administrative records from Indiana Department of Education, relevant school districts, and BGCNIC program data. Results from this RCT will be disseminated to policymakers and providers across the country to inform the replication and expansion of programs designed to support under-resourced children. We plan to enroll approximately 1,800 people in the study over three years (academic years 2024-25 to 2026-27).

Intervention

The club enrichment program intervention is an expansion of club programming designed to recover some of the learning loss of the COVID-19 pandemic, especially for disadvantaged students. Parents enroll their children in club programs through an online application, and only children between kindergarten and fifth grade are eligible for club enrichment programs.
The year-long enrichment programs are designed to use the innate passions of children to empower them to learn outside of the classroom during their time at club each day. All club activities are directed by youth development professionals (YDPs). The control group will participate in standard club programming, which consists of snack, homework time, access to therapy services, access to Beable (an online literacy-development platform), and high-yield activities including leisure reading, writing activities and games like chess or Scrabble that develop cognitive skills. The EWB treatment will have access to these same standard club program elements, but they will also attend two 45-minute sessions on SEL. The SEL plus academic treatment group will attend two 45-minute SEL sessions, one 45-minute literacy session, and one 45-minute STEM-focused session. The lesson plans for these sessions are collaboratively developed with content-expert community partners at the Robinson Learning Center and Riverbend Community Math Center.

Experimental design

The BGCNIC team iteratively designed the experimental design throughout weekly calls with the research team. The following elements describe the procedures beginning with recruitment and study enrollment.
Beginning in mid-July of each year of enrollment, BGCNIC will open their online application portal where parents/guardians can fill out an application to enroll their child (member) in after-school programming (club). This online application will contain a parental consent form, and parents consent to have their child followed in administrative records.
During each year of enrollment, standard club programming will begin in August with the academic school year. Standard club programming consists of snack, homework time, access to therapy services, access to Beable (an online literacy-development platform), and high-yield activities including leisure reading, writing activities and games like chess or Scrabble that develop young people's cognitive skills.
In early September of each year of enrollment, the research team will randomize study participants into three groups: control, SEL, and SEL-plus. Randomization odds will be approximately 33:33:33. The control group will continue with standard club programming. In addition to standard club programming, the SEL group will receive two 45-minute SEL-focused programming sessions. The SEL plus academic group will receive standard club programming, two 45-minute SEL-focused programming sessions, and two 45-minute academic-focused programming sessions (one STEM and one literacy). These separate programming sessions will occur in separate classroom spaces away from standard club programming spaces. We plan to enroll 1,800 children in the study over three years, of which approximately 600 will participate in each study group.
Through data sharing partnerships with the Indiana Department of Education (IDOE) and multiple school districts, the research team will pull academic outcomes for study participants as well as other administrative data collected by the school systems. The research team will also administer the strengths and difficulties questionnaire (SDQ) to the teachers of study participants.
Program effects will be estimated by an intent-to-treat design where standard deviation changes in grades and test scores will be regressed on a vector of observed characteristics, and a treatment group dummy variable for each treatment group. We will also report treatment-on-treated effects using a two-stage least squares regression approach. Heterogeneous impacts can be obtained by estimating the baseline regression with additional controls for groups and group-by-treatment interaction effects. These subgroups are subject to change based upon data availability as we continue to formalize data sharing relationships with relevant school districts and other stakeholders.
One-year outcomes will be collected and reported for the first enrollment cohort in summer 2025, and preliminary results will be updated and reported for each subsequent year of the study. Three-year results will be available for the first enrollment cohort in summer of 2027, and for the full sample in the summer of 2029.

We plan to investigate the following sub-groups:
1. Primary subgroup
- Baseline social-emotional well-being index: consisting of standardized T-Score categories (none to slight, mild, moderate, severe) based upon parent-report Diagnostic and Statistical Manual of Mental Disorder (DSM) cross-cutting measures for depression, anxiety, and anger.

We are also interested in exploring heterogeneity by their baseline cognitive skills index, gender, and race. To construct the cognitive skills index, we will use grades, standardized test scores, and other academic information prior to randomization to predict baseline cognitive measures.

2. Other Primary Subgroups
- Baseline cognitive measures index: We will combine grades and standardized test scores prior to randomization into an index to capture baseline cognitive/academic performance.
- Student gender
- Student race

3. Secondary Subgroups
- Baseline non-cognitive measures index: we will combine disciplinary outcomes, attendance, and grade repetition (and homework completion if available) prior to randomization to measure non-cognitive skills at baseline.
- To measure level of poverty and family resources at baseline: consisting of household income, household structure, benefits usage (SNAP/TANF). We will split the sample at the median of the baseline index for poverty level and family resources.
- Student academic grade (at point of randomization)

Design details

Randomization method
Computer (STATA)
Randomization unit
individual
Clusters
1100 individuals
Observations
1100 individuals
Arms
Approximately 370 individuals in control group, 370 individuals in the SEL treatment group, 370 individuals in the SEL plus academic treatment group
Primary outcomes
Standardized GPA, Standardized test scores, Social-emotional skills
Secondary outcomes
Chronic absenteeism, attendance, grade repetition, homework completion, disaggregated standardized test scores, behavioral incidents, teacher-reported behavior

Cited by

No citation data available for this registration yet.

Keywords