Subjective versus Objective Performance Pay for Teachers

How to cite

Brown, Christina; Andrabi, Tahir. 2019. "Subjective versus Objective Performance Pay for Teachers." AEA RCT Registry. AEARCTR-0003835. https://doi.org/10.1257/rct.3835-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Christina Brown* , Tahir Andrabi

* Primary investigator

Registry ID
AEARCTR-0003835
DOI
https://doi.org/10.1257/rct.3835-1.0
First registered
Last updated
Status
On going
Trial dates
2017-10-02 – 2019-12-31
Countries
Private
JEL codes
I2, J3, J5
Registry record
http://www.socialscienceregistry.org/trials/3835

Abstract

This study looks at the effects of providing teachers raises based on their students' test scores versus their principal's rating of them as compared to a flat raise. We measure the effect on teacher behavior in and out of the classroom, teacher retention, and student performance. While subjective performance pay is easier to implement and can be more successful at addressing multi-tasking concerns, there is a worry that principals may be biased against certain groups or show favoritism toward teachers they are socially connected to. We look for evidence of gender discrimination and favoritism toward connected teachers as well as heterogeneity in treatment effects by more or less biased principals. Finally, we introduce an additional randomization as to how often principals observe certain teachers to vary the principals' accuracy of information. We look to see if principals accurately update and if teachers in response work harder when their effort is more observable.

Intervention

A. Incentive Treatments
Control: Flat raise - All teachers will receive the same raise (about 7%) at the end of the calendar year. This is equal to about twice the cost of living inflation.

Treatment 1a: Subjective Performance Raise - All teachers will be rated by their direct supervisor (usually a principal or vice principal) on the teacher's "effort in improving students' academic performance'' on a scale from 0-100. Supervisors will give a short justification for their rating. Within the school, teachers will then be ranked according to this subjective score, placed into 5 bins and given a raise from 0-15% based on their ranking.

Treatment 1b: Subject Performance Raise - All teachers will be rated by their direct supervisor (usually a principal or vice principal) on a number of criteria determined by HR, such as timeliness, communication and professional integrity on a scale from 0-100. Supervisors will give a short justification for their rating. Within the school, teachers will then be ranked according to this subjective score, placed into 5 bins and given a raise from 0-15% based on their ranking.

Treatment 2: Objective Performance Raise - All teachers receive a raise based on percentile value-added (Barlevy and Neal, 2016) -- their students' average percentile within their lagged percentile comparison group on the standardized exam at the end of each semester. Teachers will then be ranked within the school based on this measure of value added, placed into 5 bins and receive a raise varying from 0-15%.

On average the raise percentage is the same across all treatments.

B. Observation Treatment

Control: Status quo – Teachers observed at the normal frequency by their supervisor, typically once a semester

Treatment: Frequent observation – Principals are requested to observe teachers for at least 10 minutes once every three weeks

Experimental design

We randomly assign teachers to receive a variety of incentive contracts and cross randomize with how often their manager observes them. The key specification will look at the effect of each scheme on test scores and teacher effort. We will also look at the interaction with being observed more to determine whether that increases the perceived "bite" of the subjective incentives and therefore, teacher effort in that treatment.

We will take the data to a classical principal-agent model to understand how the perceived noisiness and distortion of each incentive system helps us understand the choices teachers make in terms of the extent and direction of effort change.

In March we will conduct a survey to understand what teachers believed about each incentive system. If teachers interpreted both treatments similarly, we will collapse those two treatments.

Note: The listed design details are being pre-specified after the treatment implementation is complete but before endline data was finished being collected. PIs had reviewed midline student grade outcomes and video data prior to pre-specification.

Design details

Randomization method
Randomization was conducted live via Stata during a meeting with the implementing partner and research staff.
Randomization unit
The unit of randomization for the incentives treatment was first at the school level (Treatment 1b vs everything else) and then the randomization was at the school-grade section level. A grade section is, for example, the primary, middle or high school section within a school. The unit of randomization for the observation treatment was first as the school level, for which principals would be trained on the observation rubric and then randomization was conducted at the teacher level for which teachers would receive more frequent observations.
Clusters
There are 155 schools and 300 grade-sections.
Observations
There are 5,000 teachers and 55,000 students
Arms
Incentive Treatment -- First level of randomization: (randomized at school level): 88 schools assigned to treatment 1b, 67 schools assigned to next level of randomization Second level of randomization: (randomized at grade section level): 43 assigned to treatment 1a, 43 assigned to treatment 2, 43 assigned to control Observation Treatment -- First level of randomization (at the school level): 99 assigned to be part of the observation treatment and 56 part of the control Second level of randomization (at the teacher level): 2100 assigned to receive treatment and 2400 assigned to control
Primary outcomes
The primary outcomes of interest are student test performance, measured in January 2019, and teacher effort, measured by attendance and clock in and out time from March 2018-November 2018 and scores on the CLASS rubric from videos of classes.
Secondary outcomes
Secondary outcomes are student satisfaction and socio-emotional skills from a survey conducted in January 2019, test performance broken down by subject (English, Math, Urdu and Science) student retention, teacher time use and satisfaction from a survey conducted in March 2019.

Cited by

No citation data available for this registration yet.

Keywords