The Equilibrium Impact of AI Assisted School Choice

How to cite

Neilson, Christopher; Kapor, Adam; Zimmerman, Seth. 2018. "The Equilibrium Impact of AI Assisted School Choice." AEA RCT Registry. AEARCTR-0003367. https://doi.org/10.1257/rct.3367-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Christopher Neilson* , Adam Kapor , Seth Zimmerman

* Primary investigator

Registry ID
AEARCTR-0003367
DOI
https://doi.org/10.1257/rct.3367-1.0
First registered
Last updated
Status
Completed
Trial dates
2018-01-01 – 2018-06-30
Countries
United States of America (North East)
JEL codes
I20, D47
Registry record
http://www.socialscienceregistry.org/trials/3367

Abstract

This project studies a randomized EdTech/AI intervention that helps parents strategize in a centralized school choice setting that rewards knowledge of admissions probabilities for different submitted portfolios. Many districts have implemented centralized assignment mechanisms that reward strategic play (see e.g. Agarwal and Somaini 2018). Prior studies indicate that many households lack the information on admissions probabilities necessary for optimal play (Kapor, Neilson, and Zimmerman 2018), and that this reduces overall welfare relative to alternative assignment mechanisms that only require households to report their own preferences truthfully. However, this work also suggests that a “best-case” personalized informational intervention in which households learn true admissions probabilities could lead to welfare outcomes that beat the strategy-proof assignment mechanisms. The first goal of this study is to see whether AI assistance that provides simulations of person-specific, application-specific assignment probabilities can approximate this best-case scenario, and to understand the distributional implications of such interventions when implemented at scale.

The second goal is to understand how households in school choice markets make dynamic tradeoffs between schools they like now and access to better schools in the future. We address this question in the context of survey questions that ask students about their preferences over and understanding of this tradeoff, and an informational intervention that helps them understand how the tradeoff depends on submitted applications to the choice system.

Experimental design

Our app-based “Apply Smart” program allows households to view admissions probabilities for hypothetical applications based on outcomes from past years. This project evaluates a randomized trial in which some respondents to home surveys of parents with children entering Kindergarten and PK receive access to the Apply Smart tool while others do not. In addition to simulating admissions chances for the current school year, students applying for spots in Pre-K also receive information about how their choices about PK this year will affect their options for Kindergarten next year. The PK choice has important implications for Kindergarten choice the next year because some PKs (those that are part of PK-8 schools) give students automatic admission to the associated primary school track, while others (primarily Headstart programs that stop at PK) do not.
We will estimate specifications that take these variables as outcomes. Right-hand side variables will include indicators for treatment and a rich set of controls based on survey covariates, including elicited baseline preferences and beliefs, as well as student demographic variables. We plan to examine effect heterogeneity by a) belief errors at baseline, b) student socioeconomic status as measured by neighborhood poverty, c) elicited preference intensity, d) student neighborhood (measures of students’ options outside of choice), e) grade, and f) “need to switch,” defined by whether the PK in which a student is currently enrolled offers the option of continuing through to Kindergarten. In addition to these cuts of the data, we plan to conduct a machine learning analysis of effect heterogeneity following Chernozhukov et al. (2018).
We plan to estimate two types of specifications. The first will compare the two treatment arms to the control arm with the sample universe. The second will compare the two treatment arms (survey vs. survey +app) within the sample of individuals who complete surveys.
In the second step of this analysis we will use our experimental results as inputs to a structural model of school choice, following Kapor, Neilson, and Zimmerman (2018). The details of the model will depend on the themes that emerge from experimental analysis in part one. The overarching idea is to incorporate the effects of the intervention on belief formation, choice participation, and placement outcomes into estimation, and then to use counterfactual simulations to learn about the equilibrium effects of scaling up the policy. We will evaluate these counterfactuals using measures of welfare (typically distance-metric welfare in this literature) as well as school attributes such as test score performance for past students.
We note that, at the time of this writing, we anticipate the output from this intervention will consist of two separate papers, corresponding to the two research goals outlined in the motivation setting. The first will focus on the “static” school choice problem, and experimental specifications split by points a) through d) outlined above. The second will focus on the dynamics of household decisionmaking, and include splits by points e) and f). In particular, we anticipate that the second project will include interactions between current grade and option to continue. Given sample sizes, we anticipate these specifications will have low statistical power within the standard experimental framework but will still provide information that is informative in the model estimation step.
In addition to these two main steps, we will conduct a descriptive analysis of student preferences and beliefs.
We conducted the survey and intervention between January 2018 and March 2018. We conducted followup surveys between March 2018 and May 2018. At the time of this writing (September 2018) we have knowledge of survey/intervention takeup in the treatment and control arms. but have not begun analysis of experimental data. This statement is not verifiable. We note, however, that we outlined this same analysis in applications for grant funding at JPAL-NA submitted in February 2018, prior to data collection. We can provide this grant application upon request. Further, we think the outcomes, specifications, and sample splits outlined here represent the natural set of first-order test for the effect of informational interventions in the school choice context. With respect to sample collection, our goal for the initial survey was to conduct as many survey/treatment interventions as possible during the short window between the opening of the choice process in January 2018 and the submission of applications in March 2018. With respect to followup survey sample collection, our goal was to collect as many followup surveys as possible between the release of choice outcomes in March 2018 and the conclusion of our survey arrangement with NHPS at the end of the 2018 school year.

Design details

Randomization method
Household-level randomization within the sample of households with a student enrolled in PK3 or PK4 or who had applied to PK3/PK4 in the past but were not placed.
Randomization unit
Household
Clusters
N=1981 households in sample universe N=219 households completing either treatment arm 1 (survey only) or treatment arm 2 (survey+app) N=222 households completing followup survey
Observations
N=1981 households in sample universe N=219 households completing either treatment arm 1 (survey only) or treatment arm 2 (survey+app) N=222 households completing followup survey
Arms
Control: 671 (132 completing followup) Treatment arm 1: 649 (47 completing followup) Treatment arm 2: 661 (43 completing followup)
Primary outcomes
We will examine the effects of our intervention on the following outcomes: • Elicited beliefs about admissions probabilities and changes in these beliefs relative to baseline, as measured in a follow-up survey • Participation in choice (i.e. submitting an application) • Attributes of the submitted application (how many schools listed, admissions chances at schools listed, elicited preferences for schools listed, belief accuracy about schools listed) • Placement outcomes (any placement, attributes of placed school such as measures of school quality and stated preference for placed school, enrollment)
Secondary outcomes
Over the medium run, we will look at “persistence” outcomes as well (persistence in the placed school, the decision to participate in choice in subsequent years, and submitted applications in subsequent years) Over the long run, we will look at student achievement if and when these data become available (student test scores, absences, suspensions)

Cited by

No citation data available for this registration yet.

Keywords