Accuracy and Updating: How Labels and Initial Performance Affect Teacher Grading
How to cite
Chu, James; Li, Guirong; Rosa, Leonardo; Liu, Chengfang; Loyalka, Prashant. 2016. "Accuracy and Updating: How Labels and Initial Performance Affect Teacher Grading." AEA RCT Registry. AEARCTR-0001498. https://doi.org/10.1257/rct.1498-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Accurate grades help students, parents, teachers, and administrators allocate resources to improve student learning. Moreover, many life-altering institutional decisions are based on grades, such as whether or not students are given opportunities to pursue high performance tracks. Unfortunately, recent studies have shown that inaccuracies in grading occur when teachers rely on student labels (such as ethnicity or caste) to inform their grading. Moreover, studies have shown large inaccuracies in grading occur when teachers rely on labels about a student’s prior grades to assign grades.
With the above in mind, we have four goals in mind for this study. First, we seek to validate prior studies by examining how teacher assessments are affected by information about a student's prior performance. Specifically, we look at how labels about the prior performance of a student affect teacher grading.
Second and more importantly, we seek to examine how the effect of such labels changes as teachers grade additional work from a student. On the one hand, teachers may get more accurate when grading a given student’s work over time, in spite of labels. The hypothesis is that labels cause teachers to assign inaccurate grades at first, but these inaccuracies resolve as the teacher interacts with and becomes familiar with students over time. On the other hand, any inaccuracies could also compound over time, leading to cycles of cumulative (dis)advantage. This may occur if teachers assign inaccurate grades in the past and proceed to rely heavily on prior grades when assigning future grades. This may create a positive feedback cycle where minor inaccuracies amplify or become entrenched over time. In other words, we should be more worried about inaccurate grading in practice.
Third, it is unclear if the actual initial performance of a student also leads to inaccuracies in grading. Labels are social classifications imposed upon a student, whereas initial performance comes from an individual student him or herself. For example, at the start of a school year, a teacher will grade the first assignments of students and give them a grade. Will the teacher’s first grades create inaccuracies in how the teacher grades later assignments? Is the effect cumulative?
Fourth and finally, it is unclear how confirmatory or contradictory evidence changes inaccuracies in grading. For instance, if a student is labelled as a poor performer, but he or she displays high initial performance, how does the teacher respond?
Intervention
Each teacher receives the exact same set of four essays (written by actual seventh and eighth graders). The first (1) essay is graded blindly. After the teacher grades the essay, he or she is told that this is a “practice” essay. The teacher then receives a set of three essays ostensibly written by a particular student at the teacher’s school over the course of a semester (e.g. three essays written at three distinct time periods).
Teachers are randomly assigned to receive one of six different essay packets in a 3x2 crosscut design. These groups are defined as follows:
A. Essay #2 includes a description of the student as having prior performance in top 25 percentile
B. Essay #2 includes a description of the student as having prior performance in bottom 25 percentile
C. No label about student
X. When graded blindly in the past, essay #2 was of high quality (graded at top 25 percentile)
Y. When graded blindly in the past, essay #2 was of low quality (graded at bottom 25 percentile)
In regard to interventions A, B, and C above, we first reminded the teacher that we had assessed a large number of students at the teachers’ school (this was true, as we surveyed the students as a part of another experiment). For treatment conditions A and B, we wrote the following description on the top of essay 2: “In the previous semester we conducted a Chinese language arts test at your school. This student performed at the top [bottom] 25th percentile (treatment A [B] in Table 1 above) in your school.” Teachers in the no prior grade label group (treatment C) did not see this information.
With regard to treatment groups X and Y above, we selected high and low quality essays from a bank of essays that were graded by approximately 200 teachers in a prior pilot study. For the purposes of this experimental study, an essay that received a grade in the bottom 25th percentile of essay grades in the pilot study was considered as a “low quality” essay. An essay that received a grade in the top 25th percentile of essay grades in the pilot study was considered a “high quality” essay for the purposes of this experimental study.
Finally, essays 3 and 4 were identical across groups. No additional labels were added to Essays 3 and 4.
Experimental design
In each sample school, enumerators found Chinese language arts teachers in seventh and eighth grade (sample teachers). These teachers are asked to help grade a set of essays. However, teachers receive different essay packets depending on their treatment group. Teachers are randomly assigned to one of six treatment groups:
AX. Student top 25 percentile, essay of high quality
BX. Student bottom 25 percentile, essay of high quality
CX. No label, essay of high quality
AX. Student top 25 percentile, essay of low quality
BX. Student bottom 25 percentile, essay of low quality
CX. No label, essay of low quality
Design details
Cited by
No citation data available for this registration yet.