Evaluation of Practice Makes Perfect Summer Program in New York

How to cite

Chapkis, Connor; Sullivan, James; Kroeger, Sarah; Gibbs, Chloe; Corcoran, Sean. 2021. "Evaluation of Practice Makes Perfect Summer Program in New York." AEA RCT Registry. AEARCTR-0008032. https://doi.org/10.1257/rct.8032-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Connor Chapkis* , James Sullivan , Sarah Kroeger , Chloe Gibbs , Sean Corcoran

* Primary investigator

Registry ID
AEARCTR-0008032
DOI
https://doi.org/10.1257/rct.8032-1.0
First registered
Last updated
Status
On going
Trial dates
2018-05-01 – 2023-05-01
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/8032

Abstract

This study evaluates the impact of a summer learning program for students from low-income communities in the New York City area implemented by Practice Makes Perfect (PMP). This program aims to address summer learning loss through academic enrichment, tutoring, and mentoring services. Students in the program are grouped by grade level and receive services from hired teachers five hours per day, four days per week. The research team will work with PMP and their partner schools to implement a randomized controlled trial (RCT) to generate rigorous evidence of the program’s impact on academic outcomes. The initial pilot phase of the RCT was implemented in the summer of 2018. This pilot was intended to lay the foundation for a larger-scale impact evaluation.

Experimental design

1. Administrators at each school provide a list of eligible students to PMP. Eligibility criteria at each school is determined by the
individual school administrators. For each PMP class of 20 students, the school will provide a list of at least 120 eligible students. The
required number of eligible students exceeds the number of available slots in order to account for the fact that some students will not
take-up the offer to participate in PMP and to allow for a sufficiently sized control group. In particular, having a list of at least 120
students allows the take-up rate to be 1-in-3 and still maintain a control group that is the same size as those offered the program. In
that case, 60 students would be offered PMP in order to fill 20 slots. These 60 students would constitute the treatment group, while
the remaining 60 students would be designated as the control group. This master list of eligible students that the school provides to
PMP will include information so that PMP can contact the student’s family if they are selected for a PMP slot. This is the typical
recruitment process employed by PMP in working with schools to fill slots in their summer school classrooms.

2. PMP provides LEO with a de-identified master list of eligible students and the number of available spots for each school. A separate
list will be provided for each PMP class of 20 students. This de-identified list will include a student ID number, gender, age, and a
family ID to identify siblings.

3. LEO will randomly assign the order of students on the master list for each class to create a randomly ordered waitlist. LEO will then
provide PMP with the student IDs for those at the top of this waitlist.

4. PMP, working with schools, will then offer slots in the program to students in sequential order from this waitlist. If PMP has invited
all students on the list, but still has unfilled slots in the program, LEO will provide PMP with additional student IDs from the randomly
ordered waitlist.

Design details

Randomization method
Upon receiving the lists of eligible students in the appropriate grade levels from each feeder school, the researchers will assign the students to a wait list, specific to each summer school program, in a random order. The research team will provide small batches of ordered student names to PMP staff so they can conduct outreach to recruit students for the program until all spots were filled. Those students at the bottom of the wait list whose names were not transferred to PMP for recruitment comprise the control group.
Randomization unit
Randomization will occur at the individual student level, although separate random wait lists will be generated by grade level and school.
Clusters
480 students across 4 schools
Observations
480 students
Arms
240 students will receive treatment status, and 240 students will receive control status
Primary outcomes
Two indices will be constructed as our primary outcomes: an education index based on test scores and related indicators, and an engagement index based on variables such as school attendance, disciplinary actions, on-time grade promotion, and other related measures.

Cited by

No citation data available for this registration yet.

Keywords