Crowdsourcing surveys
How to cite
Galashin, Mikhail. 2022. "Crowdsourcing surveys." AEA RCT Registry. AEARCTR-0010095. https://doi.org/10.1257/rct.10095-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Researchers often ask members of the target population which evidence to collect. This includes asking which treatment arms to test, which questions to ask, and which response options to offer. Moreover, there is often a well defined goal that the researcher would want to optimize, such as the treatment effect, the predictive power of the question for some deep variable of interest, or the probability that the suggested survey options would be sufficient to classify all the respondents. However, there is little systematic evidence of the effects of such elicitation on the quality of the research design. We propose an experimental design to study this question: we first elicit suggestions for the best design of the evidence and then run a small version of the study for each of the elicited designs. This allows us to estimate the effect of monetary incentives and respondent characteristics on the design quality.
Intervention
Our experiment has two levels of intervention. On the first level, the respondents forecasting the best list of options to close an open-ended question are offered a piece-rate payment for the fraction of the second-stage respondents who use the options they suggested. On the second level, we randomize the options created by forecasting respondents to the second stage respondents.
We are interested in the effect of incentives on answer quality. We also view the characteristics of the forecaster as a source of treatment heterogeneity (Heiler and Knaus, 2022) and we are interested in estimating the characteristics associated with high treatment (option set) quality.
Experimental design
We run two sets of anonymous surveys on Cloudresearch panel of Amazon Mechanical Turk workers.
TThe surveys will proceed as follows:
Survey 1 (also called Stage 1, the main survey) :
Part A: Common questions about background, attention, and general knowledge about the topics of the survey.
Part B:
Section 1: the subjects are asked to write answers that other survey takers will use in response to two questions of interest on topics including household finance, public finance, and political economy. The two questions are drawn randomly from 4 possible questions: on. The questions are based on the open-ended questions used in the literature or ask about the reasons to select particular options in established closed questions.
The subjects are randomly assigned to one of two conditions. The first group is paid a bonus for the quality of their answers to the first question. The second is paid for the quality of their answers to the second question. The bonus is 20 cents per each person out 10 who uses an answer from the provided set as opposed to selecting "Other / my option is no listed"
Section 2: The subjects are asked to summarise their own opinion on the question in one sentence.
Section 3: The respondents are asked to predict the performance of answers of others and bonus payments are assigned for their guesses according to a scheme established in the literature (a random option is selected and the respondent is paid if her answer is close enough to the truth).
Section 4: The respondent is asked to predict the quality of her own answers.
Part C: Common questions on the demographic characteristics and general views of the respondent
Survey 2 (also called Stage 2, test survey):
Part A: Common questions about background, attention, and general knowledge about the topics of the survey
Part B: Specific to Stage 2.
First, we ask the respondents to summarise their responses in one sentence.
Second, we use the lists of answers from the respondents from the first survey to poll a new set of respondents. For each list presented to them, the new respondents are asked which of the answers best suits their beliefs, if any. The number of respondents who find their most preferred option in the list serves as the measure of the quality of the first-stage (main) survey answers. Each respondent is asked to consider 12 option lists for 2 questions.
Part C: Common questions on the demographic characteristics and general views of the respondent.
Design details
Cited by
No citation data available for this registration yet.