Effects of monitoring on higher education instructional quality and effects on student performance
How to cite
Phipps, Aaron; Podojil, Brandon; Gardiner, Ashley; Player, Daniel. 2022. "Effects of monitoring on higher education instructional quality and effects on student performance." AEA RCT Registry. AEARCTR-0009011. https://doi.org/10.1257/rct.9011-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This experiment is designed to determine whether and how instructors in higher education respond to monitoring, and how those changes affect student outcomes. Substantial research over the last two decades has established the importance of instructor quality in determining the outcomes of students in K-12 education. Studies employing rich longitudinal data have shown that these differences in instructor quality have implications beyond just contemporaneous test scores; they translate into meaningful differences in long-term life outcomes. For a number of reasons, including the lack of standardized assessments and curricula, large-scale causal evidence around the importance of instructor quality in higher education lags that of K-12 education. However, a few key quasi-experimental studies suggest that instructor quality plays a meaningful role in the college students’ success.
While a strong literature exists exploring what practices lead to greater success in the college classroom, little is known about what policies can be implemented at the administrative level to improve classroom practices. We attempt to fill this gap by studying the effects of monitoring and feedback on instructor quality and student outcomes. This study is the first to our knowledge to develop and assess the quality of an in-class observation rubric for higher education. Furthermore, the randomization of student assignment to instructors and instructor assignment to treatment provide further opportunities to test the causal effect of instructors on student performance on standardized tests and in other classes. Finally, the research design directly measures how teacher practices may change over the course of a semester as a result of monitoring and feedback. Lesson schedules are standardized across instructors, as are assignments and due dates. This standardization further allows us to assess changes in student performance directly before and after instructor observations. As a result, the design also allows us to separate how instructors are learning as a result of their feedback as opposed to how they are preparing differently in response to the possibility of an observation.
Intervention
The intervention is to monitor instructor performance in higher education. Monitoring has the potential to affect instructor effort in the expectation of being observed, as well as their performance after being observed and receiving feedback. The design allows us to parse these effects out and determine whether in-class observations provide a viable tool for administrators to improve instructor quality and student performance in higher education.
Experimental design
We plan to randomly assign instructors to receive three in-class observations either in the Fall semester or the Spring semester. As a result, we have a randomly selected sample for treatment in both semesters. Randomization will be stratified based on instructor experience, and it will be conducted using a computer random-number generator. We expect a total of about 30 different instructors per year with two semesters of teaching each. Class sizes are standardized, as are tests and lesson pacing.
The research design allows us to identify the effect of in-class observations on instructor practice and student learning in several unique ways. First, given the control group, we can assess the overall effect of observation on student performance on standardized exams. Second, we can identify how instructors improve from one observation to the next. Third, we can assess the inter-rater reliability of our observation rubric. Fourth, we can measure whether and the extent to which in-class observations have cumulative effects on student outcomes.
To measure our outcomes of interest, we will rely on (1) instructor scores from observations to test for instructor improvement, (2) student performance data retrieved from administrative data sources, and (3) student assessments of instructor quality. The administrative data includes individual scores on course assignments and summary information on overall academic performance.
Treatment is three in-class observations that occur during three specific windows in the semester. The exact day of each visit is unknown to instructors. Observers will use a specific observation rubric designed based on existing observation literature. We will then use student performance measures from administrative data that contains student performance in all their current and prior courses.
Design details
Cited by
No citation data available for this registration yet.