Effects of monitoring on higher education instructional quality and effects on student performance

How to cite

Phipps, Aaron; Podojil, Brandon; Gardiner, Ashley; Player, Daniel. 2022. "Effects of monitoring on higher education instructional quality and effects on student performance." AEA RCT Registry. AEARCTR-0009011. https://doi.org/10.1257/rct.9011-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Aaron Phipps* , Brandon Podojil , Ashley Gardiner , Daniel Player

* Primary investigator

Registry ID
AEARCTR-0009011
DOI
https://doi.org/10.1257/rct.9011-1.0
First registered
Last updated
Status
On going
Trial dates
2021-08-09 – 2024-06-01
Countries
Private
JEL codes
I23, J41, M55
Registry record
http://www.socialscienceregistry.org/trials/9011

Abstract

This experiment is designed to determine whether and how instructors in higher education respond to monitoring, and how those changes affect student outcomes. Substantial research over the last two decades has established the importance of instructor quality in determining the outcomes of students in K-12 education. Studies employing rich longitudinal data have shown that these differences in instructor quality have implications beyond just contemporaneous test scores; they translate into meaningful differences in long-term life outcomes. For a number of reasons, including the lack of standardized assessments and curricula, large-scale causal evidence around the importance of instructor quality in higher education lags that of K-12 education. However, a few key quasi-experimental studies suggest that instructor quality plays a meaningful role in the college students’ success.

While a strong literature exists exploring what practices lead to greater success in the college classroom, little is known about what policies can be implemented at the administrative level to improve classroom practices. We attempt to fill this gap by studying the effects of monitoring and feedback on instructor quality and student outcomes. This study is the first to our knowledge to develop and assess the quality of an in-class observation rubric for higher education. Furthermore, the randomization of student assignment to instructors and instructor assignment to treatment provide further opportunities to test the causal effect of instructors on student performance on standardized tests and in other classes. Finally, the research design directly measures how teacher practices may change over the course of a semester as a result of monitoring and feedback. Lesson schedules are standardized across instructors, as are assignments and due dates. This standardization further allows us to assess changes in student performance directly before and after instructor observations. As a result, the design also allows us to separate how instructors are learning as a result of their feedback as opposed to how they are preparing differently in response to the possibility of an observation.

Intervention

The intervention is to monitor instructor performance in higher education. Monitoring has the potential to affect instructor effort in the expectation of being observed, as well as their performance after being observed and receiving feedback. The design allows us to parse these effects out and determine whether in-class observations provide a viable tool for administrators to improve instructor quality and student performance in higher education.

Experimental design

We plan to randomly assign instructors to receive three in-class observations either in the Fall semester or the Spring semester. As a result, we have a randomly selected sample for treatment in both semesters. Randomization will be stratified based on instructor experience, and it will be conducted using a computer random-number generator. We expect a total of about 30 different instructors per year with two semesters of teaching each. Class sizes are standardized, as are tests and lesson pacing.

The research design allows us to identify the effect of in-class observations on instructor practice and student learning in several unique ways. First, given the control group, we can assess the overall effect of observation on student performance on standardized exams. Second, we can identify how instructors improve from one observation to the next. Third, we can assess the inter-rater reliability of our observation rubric. Fourth, we can measure whether and the extent to which in-class observations have cumulative effects on student outcomes.

To measure our outcomes of interest, we will rely on (1) instructor scores from observations to test for instructor improvement, (2) student performance data retrieved from administrative data sources, and (3) student assessments of instructor quality. The administrative data includes individual scores on course assignments and summary information on overall academic performance.

Treatment is three in-class observations that occur during three specific windows in the semester. The exact day of each visit is unknown to instructors. Observers will use a specific observation rubric designed based on existing observation literature. We will then use student performance measures from administrative data that contains student performance in all their current and prior courses.

Design details

Randomization method
Randomization conducted by computer. Instructors are randomized into a treatment semester, and the timing of observations is randomly determined as well.
Randomization unit
Randomization occurs at two levels. First, instructors are randomly assigned to be treated or not treated. Treated instructors will receive 3 unannounced in-class observations throughout their treated semester. For instructors in the treatment group, the timing of the three observations is randomly determined within pre-specified windows of observation. Treatment is clustered in the sense that all students of an instructor are in a cluster. Randomization was also blocked based on instructor years of experience.
Clusters
30
Observations
1200
Arms
600 treatment, 600 control
Primary outcomes
We will examine how monitoring and instructor quality affect the following outcomes: 1. Student test and assignment performance 2. Student evaluation of the course and of their specific instructor 3. Correlation between evaluation score, student course evaluations, and student performance 4. Individual instructor improvements between courses as assessed by in-class observations and student performance
Secondary outcomes
1. Student success in other courses outside the one observed 2. Student success in follow-on courses that rely on the material learned in the class observed 3. Student success following graduation 4. Student preferences for switching majors and the correlation with instructor quality and instructor improvement

Cited by

No citation data available for this registration yet.

Keywords