Reducing impact of student bias on feedback surveys: A pilot study

How to cite

Genetin, Brandon; Kalish, Alan; Kogan, Vladimir; Chen, Joyce. 2020. "Reducing impact of student bias on feedback surveys: A pilot study." AEA RCT Registry. AEARCTR-0006868. https://doi.org/10.1257/rct.6868-1.2000000000000002

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Brandon Genetin* , Alan Kalish , Vladimir Kogan , Joyce Chen

* Primary investigator

Registry ID
AEARCTR-0006868
DOI
https://doi.org/10.1257/rct.6868-1.2000000000000002
First registered
Last updated
Status
On going
Trial dates
2021-01-11 – 2021-07-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/6868

Abstract

This project utilizes a randomized controlled trial to assess the efficacy of utilizing modified introductory language to mitigate implicit bias in student evaluations of instruction. So-called "cheap talk" scripts are presented to survey respondents in advance and describe the hypothetical biases that tend to arise in the study context (Cummings and Taylor, 1999). This is potentially a highly cost-effective strategy to improve the quality of information generated by student evaluations of instruction, while also minimizing inequities for under-represented populations.

Intervention

All interventions related to the study will be administered by the investigators via the standard online platform for student evaluations of instruction. A subset of student participants will be randomly selected to receive additional introductory text related to the research study prior to completing the usual online evaluation of instruction for their enrolled courses. Instructor participants will not be asked to complete any activities for the study, other than to give their consent to participate. Data collected as part of the intervention will also be linked to student records (race, gender, course grade, major, rank) and instructor records (race, gender) to assess whether the effects of the intervention are heterogeneous across groups.

Experimental design

This research will utilize a “survey experiment” design, which involves embedding randomized text within a survey given to subjects. While all subjects will see the same questions (the current questions asked on the OSU Student Evaluation of Instruction), the introductory paragraph will vary across respondents.

Design details

Randomization method
Randomization occurs electronically through STATA.
Randomization unit
Randomization occurs within each course section.
Clusters
Currently one university.
Observations
Approximately 6,000 observations.
Arms
Approximately 1,500 observations for each treatment arm.
Primary outcomes
The Student Evaluation of Instructor (SEI) instrument to measure the efficacy of the instructor's teaching. The SEI is broken into ten questions: 1. The subject matter of this course was well organized 2. The instructor is well prepared 3. The instructor communicated the subject manner clearly 4. The instructor was genuinely interested in teaching 5. The instructor was genuinely interested in helping students 6. The instructor created an atmosphere conducive to learning 7. The course was intellectually stimulating 8. The instructor encouraged students to think for themselves 9. I learned a great deal from the instructor 10. Overall, I would rate this instructor as … [Poor, Fair, Neutral, Good, Excellent] Responses on questions 1 through 9 range from Strongly Disagree to Strongly Agree.

Cited by

No citation data available for this registration yet.

Keywords