Heterogeneity of Experimental Findings: Evidence from Real-Effort Tasks
How to cite
DellaVigna, Stefano; Pope, Devin. 2018. "Heterogeneity of Experimental Findings: Evidence from Real-Effort Tasks." AEA RCT Registry. AEARCTR-0002987. https://doi.org/10.1257/rct.2987-3.1
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Replication of experimental results is a focus of the recent literature. More broadly, it is important to know how experimental results vary when the context, or design, changes, that is, how heterogeneous the findings are. For example, how sensitive experimental results are to the demographics of the sample? How do they depend on the output measure used? How critical is the experimental protocol? We consider these questions in the context of a real effort task. We run variants of the same experimental treatments, varying one-by-one the demographic group considered, the task used, the measure of effort, and whether subjects know that the task is an experiment. We examine how much the experimental results change in response to these changes, compared to a pure replication.
Intervention
The interventions are 16 different treatments altering the motivation for effort and are borrowed from DellaVigna and Pope (REStud 2018). The emphasis of the paper is how the impact of the different effort motivators varies when the design of the experiment is changed with different versions.
Experimental design
Subjects will choose to participate in this study by selecting it on Amazon's Mechanical Turk service. Before the participants choose to participate, they will be provided with a brief description of the study (a” typing task”) which also tells a guaranteed flat-pay for successful submission ($1) and a time estimate for completion. Once in the task, participants are assigned to one of four versions, and within each version they are directed to complete a task with randomly-selected incentive structures.
Version 1 is a direct replication of the task used by DellaVigna and Pope (REStud 2018): participants press two alternating keyboard buttons ('a' and 'b') within a given time period (10 minutes). There are 16 different randomly-assigned treatments with varying levels and types of incentives. For example, some participants will be given bonus payments based on points scored, others will raise money for charity, and some will receive no bonus. All participants are clearly informed of their incentive structure and bonus opportunities before playing their task. There is no deception at any point in this task.
Version 2 is similar to Version 1 except that the task consists of coding the occupation from World War II historical cards for 10 minutes. As in version 1, the participants are randomized in one of 16 treatments with varying levels and types of incentives.
Version 3 involves coding of WWII cards, as in Version 2. However, this time all subjects are required to code 40 cards as their main task. After the coding of the 40 cards is completed, subjects are randomized into 16 different treatments, parallel to the ones used in Version 1 and 2. The treatments consist of different incentives offered to the participants in case they agree to code additional WWII cards, up to 20 extra cards.
Version 4 is the same as version 3, except that, unlike in Versions 1-3, there is no consent form prior to participating in the tasks, so subjects do not know that the task is an experiment. (Notice that the participants are coding historical data as part of an economic history project, so this is a data coding assignment.)
Upon completion of the task, participants will be thanked for their contribution and the flat payment of $1, along with any additional money won, will be paid within the designated time period for their treatment (typically within 24 hours).
The final sample will exclude subjects that:
(1) do not complete the MTurk task within 30 minutes of starting;
(2) exit and then re-enter the task as a new subject (as these individuals might see multiple treatments);
(3) are not approved for any other reason (e.g. they did not having a valid MTurk ID);
(4) In version 1 (a-b typing) do not complete a single effort unit; there is no need for a parallel requirement for version 2 since the participants have to code a first card to start the task;
(5) in version 1 scored 4000 or more a-b points (since this would indicate cheating);
(6) in version 2 coded 120 or more cards with accuracy below 50% (since this would indicate cheating);
(7) in versions 3 and 4 completed the 40 required cards in less than 3 minutes with accuracy below 50%, or completed the 20 additional cards in less than 1.5 minutes with accuracy below 50% (since this would indicate cheating).
Restrictions (1)–(5) are exactly the same as in the DellaVigna-Pope (REStud) experiment; restrictions (6)-(7) are meant to parallel restriction (5) for the new task. We expect that restrictions (3)-(7) are likely to be binding in only a small number of cases. For example, in a pilot of 400 subjects we did not have anyone in categories (3)-(7).
The average score in each of these treatments and versions will form the basis for the elicitation of expert forecasts.
Design details
Materials
Cited by
No citation data available for this registration yet.