Teacher observation, teacher effectiveness, and pupil attainment: An RCT in England’s secondary schools

How to cite

Burgess, Simon; Taylor, Eric; Rawal, Shenila. 2016. "Teacher observation, teacher effectiveness, and pupil attainment: An RCT in England’s secondary schools." AEA RCT Registry. AEARCTR-0001010. https://doi.org/10.1257/rct.1010-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Simon Burgess* , Eric Taylor , Shenila Rawal

* Primary investigator

Registry ID
AEARCTR-0001010
DOI
https://doi.org/10.1257/rct.1010-2.0
First registered
Last updated
Status
Completed
Trial dates
2013-12-01 – 2017-07-01
Countries
United Kingdom of Great Britain and Northern Ireland ()
Registry record
http://www.socialscienceregistry.org/trials/1010

Abstract

We study a program of informal, peer performance evaluation among classroom teachers who work in the same school. Maths and English teachers who teach these subjects in years 10 and 11 in “treatment” schools, all of which are secondary schools in areas of high deprivation in England, are asked to implement a program of peer observation for a period of two school years. The experiment is designed to estimate the effect of peer observation on teacher performance in the classroom, as measured by teachers’ contributions to student achievement and attainment, and by teachers’ observed teaching practices. The experiment is also designed to estimate different effects for teachers who are the “observer” and teachers who are the “observee”, and the influence of the frequency of observations. Observer teachers conduct an observation of their peer observee teacher and provide an evaluation; observers assign scores on specific skills using a structured rubric. However, the program is not a formal evaluation; there are no formal incentives or stakes attached to the scores.

Experimental design

Of the 82 schools in our study sample, 41 were randomly assigned to the treatment condition and 41 to a business as usual control. Random assignment occurred within blocks formed by two school-level measures: racial/ethnic composition and student test score growth attributable to the school (school value-added). As detailed below, additional randomization of teacher roles and the frequency of observations occurred within the 41 treatment schools.

In treatment schools, English and Maths teachers are asked to implement a program of peer observation and performance description for a duration of two school years. However, the program is not a formal evaluation. No explicit incentives or penalties are attached to teachers’ scores.

Within treatment schools, English and Maths teachers were randomly assigned to one of three role conditions: (i) teachers were assigned to the “observer” role with probability 1/3, (ii) to the “observee” role with probability 1/3, or (iii) to participate in both “observer” and “observee” roles with probability 1/3. Assignment to role was within school-by-subject blocks, where subject is either Maths or English.

Throughout the school year, “observers” periodically spend time watching “observees” teach in the observees’ classes. Each observation lasts 20-30 minutes. During each visit, observers are asked to pay particular attention to the observee’s performance in several specific teaching skills, and score those skills using an evaluation rubric and tablet computer program. Each skill is first scored as being “Ineffective (1-3)”, “Basic (4-6)”, “Effective (7-9)”or “Highly Effective (10-12)”.. The rubric provides a concrete description of what an observer should see happening in the classroom to warrant allocation to each one of these categories. After choosing one of these four categories, observers can choose a numeric score from within each of these categories creating a final 12 point score scale. Observers and observees were encouraged to meet and discuss the observation results however the form and nature of these feedback sessions were not prescribed. Note given the assignment of roles, some teachers will be the observer in one pair interaction, and the observee in a different interaction.

In treatment schools, English and Maths departments were randomly assigned to a “high frequency” observation condition or to a “low frequency” condition. In half of treatment schools the English department was assigned to “high frequency” and the Maths department to “low frequency.” In half of schools the department assignments were reversed. In the high-frequency condition, observees are required to be observed 12 times per year. In the low-frequency condition, the observees are to be observed 6 times per year.

These design features create three key “treatment effect” estimates of interest. First, the broad contrast of student achievement outcomes between the (i) peer observation treatment schools, and (ii) business-as-usual control schools. Second, the contrasts in teacher performance—as measured by student test scores—between (i) teachers in the observer role; (ii) teachers in the observee role; (iii) teachers in treatment schools with no role, but who might have gained through spillovers; and (iv) teachers in control schools with no role and no exposure to treatment. Third, the contrast in teacher performance between teachers who observed or were observed with (i) high frequency, or (ii) low frequency. Moreover, the design also permits estimates of the interactions between these three main contrasts.

Design details

Randomization method
In office by a computer.
Randomization unit
Please see description of Experimental Design.
Clusters
The extensive margin of treatment is assigned at the school level. There are 82 schools (41 treatment, 41 control). Since all other clusters are nested within school, this is the level of clustering most relevant for statistical inference. The frequency of peer observation is assigned at the department level within each treatment school. There are 82 clusters in this case as well: 41 treatment schools X 2 departments per school. The observer/observee conditions are assigned at the teacher level within each department and school. There are approximately 600 teachers: on average 7 teacher per department X 2 departments X 41 schools.
Observations
We observe outcome measures at two levels: teacher and student. The study sample includes approximately 25,000 students and 1,200 teachers.
Arms
41 treatment schools and 41 control schools. Within treatment schools, 41 "high frequency" departments and 41 "low frequency" departments. Within treatment school departments, approximately 200 observers, 200 observees, and 200 taking both observer and observee roles; and in control schools approximately 600 teachers without a role.
Primary outcomes
Teacher performance, as measured by teachers’ contributions to student achievement and teachers’ observed teaching practices in the classroom.

Papers from this trial

  1. Teacher Peer Observation and Student Test Scores: Evidence from a Field Experiment in English Secondary Schools. Journal of Labor Economics Volume 39, Number 4 October 2021 https://doi.org/10.1086/712997
  2. Teachers’ use of class time and student achievement Economics of Education Review, 2023, vol. 94, issue C https://doi.org/10.1016/j.econedurev.2023.102405

Cited by

No citation data available for this registration yet.

Keywords