Crowdsourcing surveys

How to cite

Galashin, Mikhail. 2022. "Crowdsourcing surveys." AEA RCT Registry. AEARCTR-0010095. https://doi.org/10.1257/rct.10095-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Mikhail Galashin*

* Primary investigator

Registry ID
AEARCTR-0010095
DOI
https://doi.org/10.1257/rct.10095-1.0
First registered
Last updated
Status
In development
Trial dates
2022-10-19 – 2022-11-07
Countries
Private
JEL codes
D83, D84, D82, B41, D90, O31
Registry record
http://www.socialscienceregistry.org/trials/10095

Abstract

Researchers often ask members of the target population which evidence to collect. This includes asking which treatment arms to test, which questions to ask, and which response options to offer. Moreover, there is often a well defined goal that the researcher would want to optimize, such as the treatment effect, the predictive power of the question for some deep variable of interest, or the probability that the suggested survey options would be sufficient to classify all the respondents. However, there is little systematic evidence of the effects of such elicitation on the quality of the research design. We propose an experimental design to study this question: we first elicit suggestions for the best design of the evidence and then run a small version of the study for each of the elicited designs. This allows us to estimate the effect of monetary incentives and respondent characteristics on the design quality.

Intervention

Our experiment has two levels of intervention. On the first level, the respondents forecasting the best list of options to close an open-ended question are offered a piece-rate payment for the fraction of the second-stage respondents who use the options they suggested. On the second level, we randomize the options created by forecasting respondents to the second stage respondents.
We are interested in the effect of incentives on answer quality. We also view the characteristics of the forecaster as a source of treatment heterogeneity (Heiler and Knaus, 2022) and we are interested in estimating the characteristics associated with high treatment (option set) quality.

Experimental design

We run two sets of anonymous surveys on Cloudresearch panel of Amazon Mechanical Turk workers.

TThe surveys will proceed as follows:
Survey 1 (also called Stage 1, the main survey) :
Part A: Common questions about background, attention, and general knowledge about the topics of the survey.
Part B:
Section 1: the subjects are asked to write answers that other survey takers will use in response to two questions of interest on topics including household finance, public finance, and political economy. The two questions are drawn randomly from 4 possible questions: on. The questions are based on the open-ended questions used in the literature or ask about the reasons to select particular options in established closed questions.
The subjects are randomly assigned to one of two conditions. The first group is paid a bonus for the quality of their answers to the first question. The second is paid for the quality of their answers to the second question. The bonus is 20 cents per each person out 10 who uses an answer from the provided set as opposed to selecting "Other / my option is no listed"
Section 2: The subjects are asked to summarise their own opinion on the question in one sentence.
Section 3: The respondents are asked to predict the performance of answers of others and bonus payments are assigned for their guesses according to a scheme established in the literature (a random option is selected and the respondent is paid if her answer is close enough to the truth).
Section 4: The respondent is asked to predict the quality of her own answers.
Part C: Common questions on the demographic characteristics and general views of the respondent

Survey 2 (also called Stage 2, test survey):
Part A: Common questions about background, attention, and general knowledge about the topics of the survey
Part B: Specific to Stage 2.
First, we ask the respondents to summarise their responses in one sentence.
Second, we use the lists of answers from the respondents from the first survey to poll a new set of respondents. For each list presented to them, the new respondents are asked which of the answers best suits their beliefs, if any. The number of respondents who find their most preferred option in the list serves as the measure of the quality of the first-stage (main) survey answers. Each respondent is asked to consider 12 option lists for 2 questions.
Part C: Common questions on the demographic characteristics and general views of the respondent.

Design details

Randomization method
Randomization is performed with a script in advance. Then the treatments are sent to the Qualrtics survey engine by a special API web app.
Randomization unit
The incentive treatments are randomized at forecaster-question level: each forecaster gets to answer one incentivized and one unincentivized question. On the second stage, the option lists are randomized on individual level. Each respondent gets to answer 2 questions, each with
Clusters
Since the option lists are randomized to multiple second-stage respondents and each respondent answers multiple questions. To limit the spillovers between the observations and allow cross-fitting, we assign the forecasting respondents to clusters of 20 and randomize them to 3 clusters of 20 second-stage respondents. This gives us approximately 40 clusters of disjoint sets of forecasters, each evaluated by 3 clusters of second-stage respondents (120 clusters in total, 60 clusters per question topic)
Observations
We aim for 800 first-stage respondents and 2400 second-stage respondents.
Arms
400 respondents first stage respondents are incentivized for the first question, and 400 for the second question. Each option list elicited in the first stage is assigned to 30 respondents in the second stage. The option lists that are used for the prediction exercise are assigned to 750 second-stage respondents so that the true value of the predicted quality is precisely measured.
Primary outcomes
Fraction of second-stage respondents who choose one of the forecasted options as opposed to "Other / my answer is not listed"
Secondary outcomes
1) Semantic similarity of the chosen option to the open-ended responses 2) Fraction of respondents for whom the chosen option topic matches the topic of the open-ended answer 3) Characteristics of responses: characters written, topic diversity, number of words. 4) Effort spent per forecast measured by time spent. 5) Semantic distance between incentivized and unincentivized answers

Cited by

No citation data available for this registration yet.

Keywords