Information Bias in Student Ratings
How to cite
Khoury, Stephanie El. 2025. "Information Bias in Student Ratings." AEA RCT Registry. AEARCTR-0016611. https://doi.org/10.1257/rct.16611-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This pilot randomized controlled trial tests whether students evaluate identical instructional content differently when they believe it is delivered by a female versus a male instructor, providing direct evidence on gender bias in professor evaluations. Undergraduate students at Arizona State University were randomly assigned at the individual level to view the same short lesson, with the instructor’s gender varied only through the voice-over. After viewing, students rated the instructor on overall quality and difficulty. We also examine heterogeneity by students’ engagement with professor review websites. The experiment is used to document the existence and direction of bias and to motivate subsequent confirmatory analyses and policy counterfactuals in the broader project. This study is retrospectively registered in the AEA RCT Registry.
Intervention
Arms: Female-voice instructor, Male-voice instructor
What varies across arms: Only the perceived gender of the instructor, conveyed by the voice-over. The script, video, and content are identical across arms.
Delivery mode and timing: One-shot exposure during an online survey. Participants view the video and immediately report quality and difficulty of the lesson.
Target population and setting: Undergraduate students at Arizona State University completing the study online.
Experimental design
This study is a two-arm, individual-level randomized controlled trial embedded in an online survey with undergraduate students at Arizona State University. Participants are randomly assigned 1:1 to view an identical short lesson in which only the perceived gender of the instructor varies: a female-voice instructor (treatment) or a male-voice instructor (control). The script and content are held constant across arms; the manipulation is delivered via the voice-over. After viewing, respondents complete report outcome in terms of quality and difficulty.
The primary outcome is the overall quality rating of the instructor measured immediately after the video. Secondary outcomes include perceived difficulty. Randomization is implemented by the survey platform using simple individual assignment with equal probabilities; participants are unaware of the alternative condition, and enumerators have no discretion over assignment.
Analyses are intention-to-treat. The main specification estimates the difference in mean outcomes between the female-labeled and male-labeled arms using OLS. Pre-specified heterogeneity examines differences by students’ self-reported engagement with professor review websites. This trial is retrospectively registered and is used as exploratory evidence on the existence and direction of bias in evaluations.
The identical protocol was re-fielded with a new sample in November 2024 using the same materials and procedures. Results are reported separately and in pooled robustness checks.
Design details
Cited by
No citation data available for this registration yet.