Response of Output to Varying Incentive Structures on Amazon Turk

How to cite

DellaVigna, Stefano; Pope, Devin. 2015. "Response of Output to Varying Incentive Structures on Amazon Turk." AEA RCT Registry. AEARCTR-0000714. https://doi.org/10.1257/rct.714-4.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Stefano DellaVigna* , Devin Pope

* Primary investigator

Registry ID
AEARCTR-0000714
DOI
https://doi.org/10.1257/rct.714-4.0
First registered
Last updated
Status
Completed
Trial dates
2015-05-14 – 2015-09-30
Countries
United States of America (N/A, Online Survey)
JEL codes
C93, D03, M52
Registry record
http://www.socialscienceregistry.org/trials/714

Abstract

This study will add to the body of research on incentive-induced behavior by quantifying the effect of various incentives on output. While a wide range of incentives have been shown to be effective (see Campbell, 2006), comparisons between different incentive types have been rare. This study hopes to quantify some of the difference between types of incentives, particularly in non-monetary forms (such as goal setting, comparison to others, etc.). The result of this study will form the basis for the elicitation of expert forecasts as registered in "Expert Forecasts of Amazon Turk Treatments" (AEARCTR-0000731).

Intervention

After Amazon Mechanical Turk users click select our task and complete the pre-requisite survey, they will be taken to a page that contains the below text:

***

You have 5 minutes maximum to read this page. If you finish early, you may proceed to the next page at your discretion.

On the next page you will complete a simple button-pressing task. The object of this task is to alternately press the 'a' and 'b' buttons on your keyboard as quickly as possible for 10 minutes. Every time you successfully press the 'a' and then the 'b' button, you will receive a point. Note that points will only be rewarded when you alternate button pushes: just pressing the 'a' or 'b' button without alternating between the two will not result in points.

Buttons must be pressed by hand only (key bindings or automated button-pushing programs/scripts cannot be used) or task will not be approved.

Feel free to score as many points as you can.

***

At this point, depending on the treatment, precisely one of seventeen different treatments will be shown to the participant in the next sentence. These treatments are designed to provide the users with different types of incentives.

Baseline:

1. Your score will not affect your payment in any way.

Basic Incentives:

2. As a bonus, you will be paid an extra 1 cent for every 100 points that you score. This bonus will be paid to your account within 24 hours.
3. As a bonus, you will be paid an extra 10 cents for every 100 points that you score. This bonus will be paid to your account within 24 hours.
4. As a bonus, you will be paid an extra 4 cents for every 100 points that you score. This bonus will be paid to your account within 24 hours.

Paying Too Little:

5. As a bonus, you will be paid an extra 1 cent for every 1,000 points that you score. This bonus will be paid to your account within 24 hours.

Charitable Giving:

6. As a bonus, the Red Cross charitable fund will be given 1 cent for every 100 points that you score.
7. As a bonus, the Red Cross charitable fund will be given 10 cents for every 100 points that you score.

Time Preferences:

8. As a bonus, you will be paid an extra 1 cent for every 100 points that you score. This bonus will be paid to your account two weeks from today.
9. As a bonus, you will be paid an extra 1 cent for every 100 points that you score. This bonus will be paid to your account four weeks from today.

Loss Aversion:

10. As a bonus, you will be paid an extra 40 cents if you score at least 2,000 points. This bonus will be paid to your account within 24 hours.
11. As a bonus, you will be paid an extra 40 cents. This bonus will be paid to your account within 24 hours. However, you will lose this bonus (it will not be placed in your account) unless you score at least 2,000 points.
12. As a bonus, you will be paid an extra 80 cents if you score at least 2,000 points. This bonus will be paid to your account within 24 hours.

Probability-Weighting:

13. As a bonus, you will have a 1% chance of being paid an extra $1 for every 100 points that you score. Approximately one out of every 100 participants who perform this task will be randomly chosen to be paid this reward. The bonus will be put in the winner's account within 24 hours.
14. As a bonus, you will have a 50% chance of being paid an extra 2 cents for every 100 points that you score. Approximately one out of two participants who perform this task will be randomly chosen to be paid this reward. The bonus will be put in the winner's account within 24 hours.

Social Comparison:

15. Your score will not affect your payment in any way. In a previous version of this task, many participants were able to score more than 2,000 points.
16. Your score will not affect your payment in any way. After you play, we will show you how well you did relative to other participants who have previously done this task.

Task Significance:

17. Your score will not affect your payment in any way. We are interested in how fast people choose to press digits and we would like you to do your very best. So please try as hard as you can.

Gift Exchange:

18. In appreciation to you for performing this task, you will be paid a bonus of 40 cents. This bonus will be paid to your account within 24 hours. Your score will not affect your payment in any way.

Experimental design

Subjects will choose to participate in this study by selecting it on Amazon's Mechanical Turk service. Before the participants choose to participate, they will be provided with a brief description of the study; which also tells a guaranteed flat-pay for successful submission and a time estimate for completion. Once in the survey, participants will fill out a brief set of questions providing demographic information like age, sex, and education. After this, participants will be directed to complete a task with randomly-selected incentive structures. The task is to see how many times a participant can press two alternating keyboard buttons (say, 'a' and 'b') within a given time period (10 minutes). There are 18 different randomly-assigned treatments with varying levels and types of incentives. For example, some participants will be given bonus payments based on points scored, others will raise money for charity, and some will receive no bonus. All participants are clearly informed of their incentive structure and bonus opportunities before playing their task. There is no deception at any point in this task.

Upon completion of the task, participants will be thanked for their contribution and the flat payment of $1, along with any additional money won, will be given within the designated time period for their treatment (typically within 24 hours). An example of the sequence of pages subjects see is attached ("SurveyWalkThrough.pdf").

The final sample will exclude subjects that (i) do not complete the MTurk task within 30 minutes of starting or (ii) exit then re-enter the task as a new subject (as these individuals might see multiple treatments) or (iii) score 4000 or more points (as we have learned from a pilot study of ~300 participants that it is physically impossible to score more than 3500 points, so it is likely that these individuals are using bots).

The average score in each of these 18 treatments will form the basis for the elicitation of expert forecasts as registered in "Expert Forecasts of Amazon Turk Treatments". To ensure that the principal investigators can make their own forecasts and that they will not interfere in the forecasting of other experts, the principal investigators will not access the experimental results until after the first round of forecasting. Results will be stored by Michael Sheldon (at Chicago Booth) and Don Moore (at UC Berkeley).

Design details

Randomization method
Number generation will randomize participants into each treatment group subject to the guarantee of an equal number of participants in each group.
Randomization unit
Individual subject.
Clusters
n/a
Observations
The number of clusters is the same as the number of observations. 10,000 people is the ideal number of subjects planned for the study. The hope is to obtain at least 5500 subjects. The task will be kept open on Amazon Mechanical Turk until either (i) two weeks have passed or (ii) 10,000 subjects have completed the study, whichever comes first. If two weeks pass without 5500 subjects completing the task, then the task will be kept open (up to six weeks) until 5500 subjects are obtained.
Arms
It was determined that we wanted to reach a sample of at least 300 subjects per treatment for a total of 300*18 = 5400 subjects so as to attain sufficiently precise estimates of the productivity per treatment. The ideal sample that we hope to achieve is approximately twice as large as that.
Primary outcomes
The key outcome variable is the number of points scored by the subject, where each point is scored as a result of pressing 'a' then 'b'. Additionally, for each subject, we store data on their button presses over time (by second).

Papers from this trial

  1. "Predicting Experimental Results: Who Knows What?" This version: August 16, 2016 https://eml.berkeley.edu/~sdellavi/wp/expertsJul16.pdf
  2. "What Motivates Effort? Evidence and Expert Forecasts." This version: March 15, 2017. https://eml.berkeley.edu/~sdellavi/wp/BehavioralForecastsMar17withOnlApp.pdf

Cited by

No citation data available for this registration yet.

Keywords