Measuring Success in Education: The Role of Effort on the Test Itself
How to cite
Livingston, Jeffrey; Qin, Xiangdong; Sadoff, Sally; List, John A.; Gneezy, Uri. 2018. "Measuring Success in Education: The Role of Effort on the Test Itself." AEA RCT Registry. AEARCTR-0003657. https://doi.org/10.1257/rct.3657-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Standardized tests comparing educational achievements are an important policy tool. U.S. students often rank poorly on such assessments. We propose that this is due not only to differences in ability but also to differences in effort on the test itself. We experimentally show that offering U.S. students incentives to put forth effort improves test performance substantially. In contrast, Shanghai students, who are top performers on assessments, are not affected by incentives. Our findings suggest that ranking countries based on low-stakes assessments does not reflect only differences in ability, but also intrinsic motivation to perform well on the test.
Intervention
10th grade students at two high schools in the U.S.A. and four high schools in Shanghai take a 25 question mathematics test which is made up of multiple choice and free answer questions that were given on past editions of the Programme for International Student Assessment (PISA). Students are randomized into either treatment or control. Members of the treatment group is given a financial incentive that is based on their test performance. Right before taking the test, they are given an envelope with $25 in cash (or the equivalent in RMB in Shanghai), and are told that $1 will be taken away for each question that is answered incorrectly. The control group takes the test with no financial incentives.
The test is taken and graded immediately upon completion by computer, so the payments of subjects in the treatment group are processed immediately at the conclusion of the experiment.
Experimental design
The working paper linked to this registration provides full details of the experimental design (NBER working paper No. 24004). Key details are summarized below.
We recruited two two high schools in the U.S.A. and four high schools in Shanghai to participate. In each school, student subjects take a 25 minute, 25 question mathematics test which is made up of multiple choice and free answer questions that were given on past editions of the Programme for International Student Assessment (PISA). Students are randomized into either treatment or control. Members of the treatment group are given a financial incentive that is based on their test performance. Right before taking the test, they are given an envelope with $25 in cash (or the equivalent in RMB in Shanghai), and are told that $1 will be taken away for each question that is answered incorrectly. The students had no advance notice of the task they would be doing or of the financial incentives. The control group takes the test with no financial incentives.
The exam is administered by computer so scores are available immediately after the test is completed.
The main experiment was conducted in 2016 at two high schools in the U.S. and at three schools in Shanghai.
U.S. School 1 is a high performing private boarding school.
U.S. School 2 is a low performing public school.
At both of these schools, all 10th grade students were required to participate.
Shanghai schools 1 through 3 include one below-average performing school, one school with performance that is just above average, and one
school with performance that is well above average. Two classes each of 10th grade math students at schools 1 and 2 and four classes of 10th grade math students at school 3 were randomly selected to participate.
in 2018, we reran the experiment in Shanghai at schools 2 and 3 and at a new school, school 4, whose performance is also well above average.
Logistics required different randomization procedures of students into treatment or control (described in more detail below):
U.S. School 1: students were randomized into treatment or control at the individual level.
U.S. School 2: students were randomized into treatment or control at the class level.
Shanghai schools 1-3 in 2016 were randomized into treatment or control at the class level.
Shanghai schools 2-4 in 2018 were randomized into treatment or control at the individual level.
Design details
Papers from this trial
- Uri Gneezy, John A. List, Jeffrey A. Livingston, Sally Sadoff, Xiangdong Qin, Yang Xu (2017) "Measuring Success in Education: The Role of Effort on the Test Itself." NBER Working Paper No. 24004 https://www.nber.org/papers/w24004
Cited by
No citation data available for this registration yet.