Information Bias in Student Ratings

How to cite

Khoury, Stephanie El. 2025. "Information Bias in Student Ratings." AEA RCT Registry. AEARCTR-0016611. https://doi.org/10.1257/rct.16611-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Stephanie El Khoury*

* Primary investigator

Registry ID
AEARCTR-0016611
DOI
https://doi.org/10.1257/rct.16611-1.0
First registered
Last updated
Status
Completed
Trial dates
2024-07-01 – 2024-07-15
Countries
United States of America (Arizona)
JEL codes
I2, C93, D83
Registry record
http://www.socialscienceregistry.org/trials/16611

Abstract

This pilot randomized controlled trial tests whether students evaluate identical instructional content differently when they believe it is delivered by a female versus a male instructor, providing direct evidence on gender bias in professor evaluations. Undergraduate students at Arizona State University were randomly assigned at the individual level to view the same short lesson, with the instructor’s gender varied only through the voice-over. After viewing, students rated the instructor on overall quality and difficulty. We also examine heterogeneity by students’ engagement with professor review websites. The experiment is used to document the existence and direction of bias and to motivate subsequent confirmatory analyses and policy counterfactuals in the broader project. This study is retrospectively registered in the AEA RCT Registry.

Intervention

Arms: Female-voice instructor, Male-voice instructor

What varies across arms: Only the perceived gender of the instructor, conveyed by the voice-over. The script, video, and content are identical across arms.

Delivery mode and timing: One-shot exposure during an online survey. Participants view the video and immediately report quality and difficulty of the lesson.

Target population and setting: Undergraduate students at Arizona State University completing the study online.

Experimental design

This study is a two-arm, individual-level randomized controlled trial embedded in an online survey with undergraduate students at Arizona State University. Participants are randomly assigned 1:1 to view an identical short lesson in which only the perceived gender of the instructor varies: a female-voice instructor (treatment) or a male-voice instructor (control). The script and content are held constant across arms; the manipulation is delivered via the voice-over. After viewing, respondents complete report outcome in terms of quality and difficulty.

The primary outcome is the overall quality rating of the instructor measured immediately after the video. Secondary outcomes include perceived difficulty. Randomization is implemented by the survey platform using simple individual assignment with equal probabilities; participants are unaware of the alternative condition, and enumerators have no discretion over assignment.

Analyses are intention-to-treat. The main specification estimates the difference in mean outcomes between the female-labeled and male-labeled arms using OLS. Pre-specified heterogeneity examines differences by students’ self-reported engagement with professor review websites. This trial is retrospectively registered and is used as exploratory evidence on the existence and direction of bias in evaluations.

The identical protocol was re-fielded with a new sample in November 2024 using the same materials and procedures. Results are reported separately and in pooled robustness checks.

Design details

Randomization method
Qualtrics random assignment.
Randomization unit
Individual. Assignment is 1:1 across the two arms within the survey platform.
Clusters
300 students
Observations
300 students
Arms
150 students per treatment arm
Primary outcomes
Overall quality rating of the instructor.
Secondary outcomes
Overall difficulty rating of the instructor.

Cited by

No citation data available for this registration yet.

Keywords