Improving Community College Effectiveness Through Performance Incentives

How to cite

Brownback, Andy; Sadoff, Sally. 2016. "Improving Community College Effectiveness Through Performance Incentives." AEA RCT Registry. AEARCTR-0001411. https://doi.org/10.1257/rct.1411-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Andy Brownback* , Sally Sadoff

* Primary investigator

Registry ID
AEARCTR-0001411
DOI
https://doi.org/10.1257/rct.1411-2.0
First registered
Last updated
Status
In development
Trial dates
2016-01-01 – 2019-05-01
Countries
Private
JEL codes
Analysis of Education (I21)
Registry record
http://www.socialscienceregistry.org/trials/1411

Abstract

We will design and test the effect of monetary and non-monetary incentives on the achievement, persistence and graduation rates of community college students. In partnership with a large community college in Indiana, we will use a randomized controlled trial to test: (1) the effects of incentive-pay for instructors, and (2) The effect of combining instructor incentives with performance-based summer scholarships for students. These incentives will be based on student performance on standardized exams given throughout the semester.

Intervention

We will implement 3 treatments:

Treatment 1: Instructor incentives. Instructors will be paid bonuses based on the performance of their students on standardized exams.
Treatment 2: Student incentives. Students will be offered vouchers for free summer enrollment if they exceed certain thresholds on their standardized exams.
Treatment 3: Instructor and Student incentives. Both instructors and students will receive incentives based on the performance of students on standardized exams.

Experimental design

In the fall semester, we will randomize instructors to 1) receive incentives based on their students' performance (Treatment 1) or 2) control. Most instructors teach two or fewer sections. These instructors will have all sections incentivized. Instructors teaching more than two sections will have two randomly selected to be incentivized. The number of sections per instructor is subject to our limit of 100 total incentivized sections.

In the spring semester, we will add in Treatments 2 and 3. We will continue to offer instructor incentives to all instructors in Treatment 1 from the fall semester, but we will randomly select one of their sections to also receive student incentives (Treatment 3). Among the sections taught by control instructors, 25 sections, each taught by a different instructor, will now be assigned to receive incentives for the students (Treatment 2), while the others will continue as control sections.

Design details

Randomization method
Randomization will be done on a computer using STATA.
Randomization unit
We will first pool all instructors of a given course, then block on instructor demographics (gender, age, tenure) and time of day (day or evening course). We will randomize until we achieve balance across enrolled student characteristics (gender, race, age, prior GPA) where no one characteristic differs at a pre-specified p-value. We will record this p-value so that we can use randomization inference to run exact significance tests on our results.
Clusters
We plan to cluster at the instructor level with approximately 144 instructors participating over the course of 2 semesters. This accounts for 20% attrition among the 120 fall 2016 instructors. These instructors will be replaced by new instructors in the spring.
Observations
Approximately 4,800 students per semester. Some students will be observed multiple times either across different courses or across different semesters. We will run a sensitivity analysis where we drop students who serve as multiple observations.
Arms
On average, instructors teach 2 sections. These sections have, on average, about 20 students each. Fall semester: We expect our pure control to have 60 instructors with 2,400 students. We will incentivize 100 total sections for the 60 instructors assigned to treatment, yielding 2,000 students in incentivized sections and 400 students in non-incentivized sections taught by incentivized instructors. The students in non-incentivized sections will be analyzed as spillover and not pure control students. Spring semester: We will have 35 instructors with 1,400 students in the pure control. We will also have 25 instructors teaching 500 students who will receive student incentives and 500 spillover students who will not receive incentives. Finally, from the 60 instructors receiving instructor incentives before, we will randomly select 50 sections (1,000 students) to receive combined incentives with no instructor overseeing more than one of these sections. We will also randomly select 50 sections (1,000 students) to receive instructor incentives. Thus, we will have 1,000 students in the instructor incentive treatment, 1,000 in the combined incentive treatment, and 400 more spillover students.
Primary outcomes
Phase I: Includes the semesters the intervention is taking place (fall 2016, spring 2017) and enrollment in the semester subsequent to the intervention. Primary outcome: Performance on the standardized assessment assigned to treatment or control o In order to compare performance across multiple exams. We will standardize exam scores (mean 0, standard deviation 1) within course using the full population of students taking the exam during that semester. The primary unit of analysis will be standard deviation units on the exam. Secondary outcomes: o Credit accumulation during the intervention semester o Student retention into subsequent semesters (spring 2017 for fall 2016; summer 2017 and fall 2017 for spring 2017) defined as whether a student is enrolled in at least one course as of the census date. Of particular interest is the impact of the student incentives on summer enrollment and subsequent retention into the fall. o Faculty retention into subsequent semesters o Faculty preferences for incentives as measured by survey responses o Faculty time use and job satisfaction as measured by survey responses. We will correct for multiple hypothesis testing within the family of survey measures included using the method developed by List et al. (2016). o Student evaluation scores for treated and control faculty Sensitivity analysis: o For exams with both multiple choice and free response sections, compare whether the results differ in the free response section to examine the potential influence of “gaming” o Exclude students from the analysis who are exposed to multiple treatments in different courses during the 2016-2017 school year o Examine the performance of students in “leftover” sections. o Instructor and student attrition in treatment and control groups. Phase II: Includes semesters subsequent to the intervention for up to a total of six years (including the intervention year). We will collect data at the end of the fall and spring semesters each year. As these data become available they will supersede previous analysis as the primary outcome. Primary outcome: Final degree attainment as defined by the National Student Clearinghouse Secondary outcomes: Progress towards degree attainment o Enrollment at Ivy Tech defined as whether a student is enrolled in at least one course as of the census date. o Credit accumulation at Ivy Tech o Transfer to a 4-year institution o Enrollment at a 4-year institution as defined by the National Student Clearinghouse o Performance of spillover students Sensitivity analysis o Exclude students from the analysis who are exposed to multiple treatments in different courses during the 2016-2017 school year Phase III: Exploratory work to track the long-term impact of the intervention on labor market outcomes for up to six years (including the intervention year). The exact outcomes will be determined once Ivy Tech has established what data it will be collecting from the Indiana Department of Workforce Development. We would like to measure employment status and income.

Cited by

No citation data available for this registration yet.

Keywords