Perceived Anonymity and Socially Desirable Responding in Course Evaluations: Evidence from a Die-Assisted Response Design

How to cite

Irlenbusch, Bernd; Rubinstein, Yona. 2026. "Perceived Anonymity and Socially Desirable Responding in Course Evaluations: Evidence from a Die-Assisted Response Design." AEA RCT Registry. AEARCTR-0018221. https://doi.org/10.1257/rct.18221-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Bernd Irlenbusch* , Yona Rubinstein

* Primary investigator

Registry ID
AEARCTR-0018221
DOI
https://doi.org/10.1257/rct.18221-1.0
First registered
Last updated
Status
Completed
Trial dates
2026-03-27 – 2026-03-28
Countries
United Kingdom of Great Britain and Northern Ireland (London)
JEL codes
C91
Registry record
http://www.socialscienceregistry.org/trials/18221

Abstract

This study tests whether a die-assisted response option reduces socially desirable responding in student evaluations. Students answer two evaluation items on a 1-to-6 scale: one about the quality of their study program and one about the teaching quality of the course professor. For one item, participants answer directly; for the other, they may either answer directly or roll a die and report the resulting number. The assignment of the die option and the order of the two questions are counterbalanced across four questionnaire versions. The main hypothesis is that responses to questions for which the die option is available will be less socially desirable, and therefore more honest on average, because participants can attribute their response to chance. The main analysis compares ratings in die versus non-die conditions while controlling for question type and order, with heterogeneity analyses by question and robustness checks using ordinal and binary-response models.

Intervention

For the question assigned to the die condition, respondents are told that they may either answer truthfully or roll a six-sided die and report the resulting number. For the direct-response question, participants answer truthfully. Participation is voluntary and anonymous, and no names are recorded.

Experimental design

The study uses a within-subject two-question design combined with between-subject variation in question order and die assignment. Each participant answers the following two questions on a 1-to-6 scale:
• Question A: “Overall, how would you rate the quality of your study program?”
• Question B: “Overall, how would you rate the professor’s teaching in this course?”

The response scale is:
1 = very poor
2 = poor
3 = fair
4 = good
5 = very good
6 = excellent
Each participant receives one of four questionnaire versions:
• AdB: Question A first with die option; Question B second without die option
• ABd: Question A first without die option; Question B second with die option
• BdA: Question B first with die option; Question A second without die option
• BAd: Question B first without die option; Question A second with die option
Thus, order and die assignment are both counterbalanced across participants.

Design details

Randomization method
Reshuffling of instruction sheets (4 x 25) by hand.
Randomization unit
Each individual answers two question. Thus, we cluster for individuals.
Clusters
100 class participants.
Observations
100 answers without die and 100 answers with die.
Arms
100 class participants. Each one answers two questions - one with a die and one without.
Primary outcomes
The primary outcomes are the numeric ratings for: • study program quality • professor teaching quality Each outcome ranges from 1 to 6.
Secondary outcomes
A useful secondary outcome is an indicator for a relatively critical response, for example: • rating of 1–3 versus 4–6, or • rating of 1–2 versus 3–6 This can help quantify whether the die option increases the probability of a negative or critical evaluation.

Cited by

No citation data available for this registration yet.

Keywords