Performance pay for introspection
How to cite
Gray-Lobe, Guthrie; Hickman, Peter. 2021. "Performance pay for introspection." AEA RCT Registry. AEARCTR-0007425. https://doi.org/10.1257/rct.7425-1.1
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We examine the effects of different incentive schemes on effort on a subjective classification task on the labor market platform MTurk.
Subjective classification tasks are common strategies to gather information on tastes, attitudes, and develop training datasets for artificial intelligence. When the output of the task is subjective, designing effective incentive schemes is challenging because effort is difficult to observe. However, opinions are often given freely, suggesting that the cost of providing a subjective opinion is low (or even negative), so that incentivizing effort may be ineffective or even counterproductive if monitoring and incentives either crowd out altruistic motives or generate multi-tasking problems and potentially incentivize gaming.
This pilot broadly investigates how to incentivize respondents to perform a simple introspective task: classifying responses to an open-ended question according to “originality”. MTurk workers are asked to report both their first order beliefs (what they think) and their second order beliefs (what they think others think). Using several measures of respondent effort, we will compare the performance of workers under a range of different incentive schemes including fixed wage schemes, various forms of attention checks, and the Bayesian Truth Serum (BTS).
For each incentive scheme, we will examine the level of effort using a range of novel outcomes, including an incentivized measure of the degree of disutility associated with the task. We will use these measures to examine whether performance incentives for subjective tasks increases the level of effort, whether linking performance pay to particular sub-tasks creates crowds out effort on other tasks, and whether the effects of different incentive schemes are heterogeneous across people with different predispositions to exert effort when there are no performance incentives.
Intervention
We examine the effects of different incentive schemes on effort on a subjective classification task on the labor market platform MTurk.
Subjective classification tasks are common strategies to gather information on tastes, attitudes, and develop training datasets for artificial intelligence. When the output of the task is subjective, designing effective incentive schemes is challenging because effort is difficult to observe. However, opinions are often given freely, suggesting that the cost of providing a subjective opinion is low (or even negative), so that incentivizing effort may be ineffective or even counterproductive if monitoring and incentives either crowd out altruistic motives or generate multi-tasking problems and potentially incentivize gaming.
This pilot broadly investigates how to incentivize respondents to perform a simple introspective task: classifying responses to an open-ended question according to “originality”. MTurk workers are asked to report both their first order beliefs (what they think) and their second order beliefs (what they think others think). Using several measures of respondent effort, we will compare the performance of workers under a range of different incentive schemes including fixed wage schemes, various forms of attention checks, and the Bayesian Truth Serum (BTS).
For each incentive scheme, we will examine the level of effort using a range of novel outcomes, including an incentivized measure of the degree of disutility associated with the task. We will use these measures to examine whether performance incentives for subjective tasks increases the level of effort, whether linking performance pay to particular sub-tasks creates crowds out effort on other tasks, and whether the effects of different incentive schemes are heterogeneous across people with different predispositions to exert effort when there are no performance incentives.
Experimental design
The data for this study come from an mTurk subjective classification task. Workers were asked to classify reports from Kenyan pupils on possible uses of a spoon in terms of the degree of “originality”. Workers were asked to report their first order beliefs about whether the use was “original” (three-point scale), and their second order beliefs about the percentage of other mTurkers who report that the use was “original”. Each round, workers are asked to grade 100 items. Of the 100 items, 20 will repeat (10 unique items x 2 occurrences). Workers complete 2-3 100-item rounds.
Treatments vary how the task is incentivized. There are six treatment arms:
TA: Control (N=200) – fixed payment
TB: Intrinsic motivation (N=160) – fixed payment
TC: Internal consistency (N=160) - Respondents receive a bonus from classifying repeated items the same way.
TD: Group consistency (N=160) - Workers receive an incentive based on the similarity of their second order beliefs to others’ reported first order beliefs
TE: Bayesian Truth Serum (BTS) (N=160)- Workers are incentivized using the BTS mechanism
TF: Attention checks (N=160) - Workers are incentivized to pay attention by giving a pre-specified answer for items containing a pre-specified word.
Workers first complete a practice round of 10 items. They then complete round 1 with 100 items, all unincentivized.
Before round 2, workers are randomized into treatments and then complete another 100 items.
After completing the second round, workers participate in a Becker-Degroot-Marshalk auction to participate in a third round. Workers submit the lowest base wage for which they would agree to work another round under the same incentive scheme. Workers who bid an amount below a randomized offer are required to participate in the third round with the base wage being the randomized offer.
The base wage will be US$ 6, the total bonus will be up to US$ 4.
The MDE for a comparison between any two treatment arms will be 0.29 SDs (alpha 0.05, beta = 0.80).
Design details
Cited by
No citation data available for this registration yet.