Question Order Bias in a Survey on User Satisfaction with Public Services

How to cite

Pedersen, Mogens Jin; Larsen, Morten; Thau, Mads; Mikkelsen, Maria. 2018. "Question Order Bias in a Survey on User Satisfaction with Public Services." AEA RCT Registry. AEARCTR-0003326. https://doi.org/10.1257/rct.3326-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Mogens Jin Pedersen* , Morten Larsen , Mads Thau , Maria Mikkelsen

* Primary investigator

Registry ID
AEARCTR-0003326
DOI
https://doi.org/10.1257/rct.3326-2.0
First registered
Last updated
Status
Completed
Trial dates
2018-10-01 – 2018-10-22
Countries
Denmark ()
Registry record
http://www.socialscienceregistry.org/trials/3326

Abstract

Public service organizations often evaluate their performance using surveys of user satisfaction with the offered services. The proposed study examines how the order of questions in a user satisfaction survey affects reported satisfaction with specific aspects of service as well as overall satisfaction. Research reveals mixed findings on the effects of presenting questions about satisfaction with specific aspects of government performance before or after questions about overall satisfaction. Using a split-ballot survey experiment, we aim to provide new evidence that may help resolve the issue. We embed the experiment in a national survey by a government agency. The agency is responsible for managing and granting government funding for social welfare projects. Private and public organizations of all kinds and type may (and do) apply. The survey captures grant recipients’ satisfaction with the agency services relating to the processes of applying for, receiving, and administering the grant. We test the effects of two treatments on the survey respondents’ reported service satisfaction: (1) survey question on overall satisfaction presented before questions on satisfaction with specific aspects of service and (2) question on overall satisfaction presented after questions on satisfaction with specific aspects of service.

Intervention

The intervention consists of variation in the order of questions in a user satisfaction survey.

Experimental design

The study involves a two-armed, parallel-design, individually randomized trial (in the form of a split-ballot survey experiment) carried out among a sample of organizations having successfully applied for government grant funding in 2017 and 2018 (first half year) for starting a new social welfare project.

On behalf of a government agency responsible for managing and awarding government grant funding for new social welfare projects, we will send out a survey request to all organizations having successfully applied for grant funding in 2017 and 2018 (first half year). For each organization, the survey request will be send to a single employee (i.e., the person listed as contact person in the government agency’s records).

We will manipulate the order of the questions in the survey. By random assignment, survey respondents will receive one of two versions of a user satisfaction survey. The two versions will differ with respect to the order of the survey questions (but are otherwise identical):

Version 1: Survey question on overall satisfaction presented before questions on satisfaction with specific aspects of service.

Version 2: Survey question on overall satisfaction presented after questions on satisfaction with specific aspects of service.

Design details

Randomization method
Randomization is carried out by simple randomization by computer.
Randomization unit
The individual survey respondent.
Clusters
500 respondents
Observations
500 respondents
Arms
500 respondents (250 in each treatment arm)
Primary outcomes
Reported service satisfaction, overall Reported service satisfaction, specific aspects of service
Secondary outcomes
Survey completion time

Cited by

No citation data available for this registration yet.

Keywords