Technology, Information, and School Choice: A Randomized Controlled Trial

How to cite

Cohodes, Sarah; Corcoran, Sean; Jennings, Jennifer; Sattin-Bajaj, Carolyn. 2019. "Technology, Information, and School Choice: A Randomized Controlled Trial." AEA RCT Registry. AEARCTR-0003736. https://doi.org/10.1257/rct.3736-2.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Sarah Cohodes* , Sean Corcoran , Jennifer Jennings , Carolyn Sattin-Bajaj

* Primary investigator

Registry ID
AEARCTR-0003736
DOI
https://doi.org/10.1257/rct.3736-2.1
First registered
Last updated
Status
Completed
Trial dates
2016-06-01 – 2019-06-30
Countries
United States of America (New York City)
JEL codes
D83, I21, I24
Registry record
http://www.socialscienceregistry.org/trials/3736

Abstract

New York City requires all 8th grade students to engage in a school choice process to be assigned to a high school through a centralized assignment mechanism. The admissions process is complex and cognitively demanding, requiring students to select among schools with different screening methods, curricular themes, and admissions preferences. To test the potential of information to help students navigate the process, we fielded a series of informational interventions in a school-level randomized controlled trial. The initial year of our study tested the effect of a paper-based informational tool – a list of relatively high-performing high schools -- in a school-level RCT of 165 high-poverty New York City middle schools (Corcoran, Jennings, Cohodes, and Sattin-Bajaj, 2018). We found that access to the informational tool increased applications to listed schools and decreased the likelihood that students attend a high school with a graduation rate below 70%.

We extended this project for an additional two years as an school-level RCT, which expanded the number of schools served to over 450 and the types of interventions offered to include a variety of technological innovations. We contrast a paper and a digital version of a curated list of schools, a guided search tool, and a directory search tool to a control group. School personnel received the intervention tools to distribute via mail, along with supportive materials (lesson plans, video guides, and support from the study office), simulating the most likely mode of dissemination if a school district were to adopt such tools. This was in contrast to the prior year’s study where trained research assistants delivered materials in a lesson. In 2016-2017, schools were randomly assigned to one of three informational interventions or a control group. In 2017-2018, we replicated the prior years random assignment to control or treatment and offered two informational interventions in the treated groups.

Intervention

Schools were assigned to one of three informational interventions or a control group. School personnel received the intervention tools to distribute via mail, along with supportive materials (lesson plans, video guides, and support from the study office), simulating the most likely mode of dissemination if a school district were to adopt such tools. Study personnel provided telephone support to encourage material use and answer questions. The three informational interventions were:

1) Curated List of High Schools ("Fast Facts"): this group received a middle school-specific concise listing of 26 geographically proximate high schools with graduation rates above 75%, along with travel time information, the graduation rate, and information about how to apply. In cross-randomization within this group, students received either a digital or paper version of the tool. A further cross-randomization made slight variations to the high school list. A third of high school lists included a caution about two nearby high schools that had low graduation rates and a third of the lists included a caution about two nearby high schools that had low admissions rates.

2) Personalized Information about High Schools from a School Choice “App”: this group received a guided introduction to an interactive smartphone/web-based tool designed to help students translate their preferences into a list of school recommendations. The app serves as a guided search tool, prompting students to identify their current middle school and their preferences for commute time, academic interests, and extra-curricular interests. It then generates a list of schools, along with performance data, that students can save, share, and explore further.

3) General Information about High Schools from a Searchable Directory ("School Finder"): This group received a guided introduction to the NYCDOE SchoolFinder, a search engine for finding high schools launched in the 2016-17 high school admissions cycle. Since all students had access to this tool (including in the control group), this group allows us to test the effect of the supportive materials that were offered as part of the intervention.

Experimental design

For 2016-2017:
Randomization was conducted in two “tiers.” Tier 1 was made up of schools that participated in our 2015-2016 RCT. Because school counselors may have provided prior years materials to students, and for ethical reasons, we decided that all schools in Tier 1 would receive a treatment, even if they were assigned to control in 2015-2016. Thus these schools contribute to estimating contrasts across treatments, but not comparisons to the control group. We retained the 39 blocks formed by matching in the previous year. See https://www.socialscienceregistry.org/trials/2951/history/37072 and http://www.nber.org/data-appendix/w24471/w24471.appendix.pdf for details on that randomization process. Within these 39 blocks, schools were randomly assigned to one of the three Fast Facts treatments, SchoolFinder, or the App. By block, the Fast Facts schools within the block were randomly assigned to either digital only delivery or digital and paper delivery.

Randomization for Tier 2 consisted of schools new to the study in 2016-2017, and were high and medium poverty schools in NYC. The approximately 100 lowest poverty schools in NYC were excluded from the experiment. Schools were randomized regardless of their intent to participate; that is, we did not recruit schools in advance for participation. We then blocked these schools into blocks of 6 schools (were possible). With 6 schools in a block, the modal block assigned one school to each of the three Fast Facts versions, one school to School Finder, one to the App, and one to control. Blocks were thus matched sextuplets of schools selected using a Mahalanobis distance measure of difference between schools (see Bruhn & McKenzie 2009; King et al. 2007). School variables used in the matching procedure included prior choice outcomes (e.g., the mean graduation rate of first round matches in 2015-16), prior achievement (mean ELA and math scores in 2015-16), economic disadvantage (the percent of students in poverty), and school size. To maintain face validity, blocking was conducted within borough and geographically isolated schools were blocked together (i.e. the Rockaways and Staten Island). Additionally, schools were blocked within categories based on their response to recruitment for the 2015-2016 experiment so that blocks were formed within groups of schools that had similar characteristics (e.g. school has returning 8th graders, school did not choose to participate in 2015-2016, school new to study, etc.) Within these blocks, schools were randomly assigned to treatments or control. The blocks were then listed in a random order and a cross randomization that alternated Fast Facts delivery method (digital only or digital and paper) was implemented.

For 2017-2018:
In the second year of the study, we used the extant randomization described above for 2017-2018. Schools that were previously assigned to one of the Fast Facts interventions were assigned to Fast Facts again (the low-graduation version) and schools that were assigned either School Finder or the App were assigned the App. Materials were “refreshed” and delivered electronically to guidance counselors. If a school received a new counselor since the previous year’s treatment, they were sent additional paper materials.

Design details

Randomization method
computer program
Randomization unit
school
Clusters
In 2016-17: 468 schools In 2017-18: 453 schools
Observations
In 2016-17: 59,492 students In 2017-18: 58,614 students
Arms
In 2016-17: Control: 59 schools Fast Facts: 249 (123 digital delivery, 126 digital and paper delivery; 82 classic Fast Facts, 83 Fast Fasts "low-graduation", 84 Fast Facts "low-odds") School Finder: 80 App: 80 In 2017-2018: Control: 59 schools Fast Facts: 242 schools (digital delivery of the low-grad version) App: 152 schools
Primary outcomes
a. High school choice outcomes (for 1st, top 3, and all choices; matched school; and enrolled school) i. School presence on the Fast Facts and/or supplementary lists 1. presence on the list 2. presence in the top or bottom half of the list ii. High school characteristics 1. 4-year graduation rate 2. whether the graduation rate was below the Fast Facts threshold of 70% 3. travel time from the middle school to the high school 4. location in the same borough as the student’s residence 5. applications per seat in the prior year (a measure of demand) 6. admissions method (e.g. screened or limited unscreened, different types of high schools in the NYC system) 7. variability in graduation rates (the difference between the highest and lowest graduation rate of schools appearing on a student's application) 8. academic or career interest area (e.g. STEM) 9. within-application consistency in interest area, calculated as the highest percentage of choices from the same interest area i. Other admissions process and enrollment outcomes 1. number of choices submitted (up to12) 2. open house priority status for limited unscreened programs (a lottery preference given to students who attend an event for certain schools) 3. ranking of the student by screened programs 4. whether or not the student was matched to his or her 1st choice, 1st-3rd choice, or any choice in the first round 5. participation in the second round after a successful match 6. matriculation to the matched school in 9th grade. Other outcome variables b. Students’ academic progress and other outcomes in high school i. Credit accumulation in 9th-12th grade, and GPA; ii. New York State Regents exam scores and passing rates; iii. On-time graduation; iv. Engagement and perceptions of school (e.g. attendance rate and responses to the NYC DOE’s Learning Environment Survey) C. Post-high school outcomes (pending data availability) i. College enrollment, persistence, and graduation ii. Earnings iii. Arrest records iv. Voter registration, voting

Papers from this trial

  1. Cohodes, Sarah R., Sean P. Corcoran, Jennifer L. Jennings, and Carolyn Sattin-Bajaj.. “When Do Informational Interventions Work? Experimental Evidence from New York City High School Choice.” National Bureau of Economic Research Working Paper 29690, 2022. https://www.nber.org/papers/w29690

Cited by

No citation data available for this registration yet.

Keywords