Question Order Bias in a Survey on User Satisfaction with Public Services
How to cite
Pedersen, Mogens Jin; Larsen, Morten; Thau, Mads; Mikkelsen, Maria. 2018. "Question Order Bias in a Survey on User Satisfaction with Public Services." AEA RCT Registry. AEARCTR-0003326. https://doi.org/10.1257/rct.3326-2.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Public service organizations often evaluate their performance using surveys of user satisfaction with the offered services. The proposed study examines how the order of questions in a user satisfaction survey affects reported satisfaction with specific aspects of service as well as overall satisfaction. Research reveals mixed findings on the effects of presenting questions about satisfaction with specific aspects of government performance before or after questions about overall satisfaction. Using a split-ballot survey experiment, we aim to provide new evidence that may help resolve the issue. We embed the experiment in a national survey by a government agency. The agency is responsible for managing and granting government funding for social welfare projects. Private and public organizations of all kinds and type may (and do) apply. The survey captures grant recipients’ satisfaction with the agency services relating to the processes of applying for, receiving, and administering the grant. We test the effects of two treatments on the survey respondents’ reported service satisfaction: (1) survey question on overall satisfaction presented before questions on satisfaction with specific aspects of service and (2) question on overall satisfaction presented after questions on satisfaction with specific aspects of service.
Intervention
The intervention consists of variation in the order of questions in a user satisfaction survey.
Experimental design
The study involves a two-armed, parallel-design, individually randomized trial (in the form of a split-ballot survey experiment) carried out among a sample of organizations having successfully applied for government grant funding in 2017 and 2018 (first half year) for starting a new social welfare project.
On behalf of a government agency responsible for managing and awarding government grant funding for new social welfare projects, we will send out a survey request to all organizations having successfully applied for grant funding in 2017 and 2018 (first half year). For each organization, the survey request will be send to a single employee (i.e., the person listed as contact person in the government agency’s records).
We will manipulate the order of the questions in the survey. By random assignment, survey respondents will receive one of two versions of a user satisfaction survey. The two versions will differ with respect to the order of the survey questions (but are otherwise identical):
Version 1: Survey question on overall satisfaction presented before questions on satisfaction with specific aspects of service.
Version 2: Survey question on overall satisfaction presented after questions on satisfaction with specific aspects of service.
Design details
Cited by
No citation data available for this registration yet.