Measuring Success in Education: The Role of Effort on the Test Itself

How to cite

Livingston, Jeffrey; Qin, Xiangdong; Sadoff, Sally; List, John A.; Gneezy, Uri. 2018. "Measuring Success in Education: The Role of Effort on the Test Itself." AEA RCT Registry. AEARCTR-0003657. https://doi.org/10.1257/rct.3657-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jeffrey Livingston* , Xiangdong Qin , Sally Sadoff , John A. List , Uri Gneezy

* Primary investigator

Registry ID
AEARCTR-0003657
DOI
https://doi.org/10.1257/rct.3657-1.0
First registered
Last updated
Status
Completed
Trial dates
2015-11-04 – 2018-12-10
Countries
China (Shanghai), United States of America (Massachusetts)
JEL codes
C91, D12, D81
Registry record
http://www.socialscienceregistry.org/trials/3657

Abstract

Standardized tests comparing educational achievements are an important policy tool. U.S. students often rank poorly on such assessments. We propose that this is due not only to differences in ability but also to differences in effort on the test itself. We experimentally show that offering U.S. students incentives to put forth effort improves test performance substantially. In contrast, Shanghai students, who are top performers on assessments, are not affected by incentives. Our findings suggest that ranking countries based on low-stakes assessments does not reflect only differences in ability, but also intrinsic motivation to perform well on the test.

Intervention

10th grade students at two high schools in the U.S.A. and four high schools in Shanghai take a 25 question mathematics test which is made up of multiple choice and free answer questions that were given on past editions of the Programme for International Student Assessment (PISA). Students are randomized into either treatment or control. Members of the treatment group is given a financial incentive that is based on their test performance. Right before taking the test, they are given an envelope with $25 in cash (or the equivalent in RMB in Shanghai), and are told that $1 will be taken away for each question that is answered incorrectly. The control group takes the test with no financial incentives.

The test is taken and graded immediately upon completion by computer, so the payments of subjects in the treatment group are processed immediately at the conclusion of the experiment.

Experimental design

The working paper linked to this registration provides full details of the experimental design (NBER working paper No. 24004). Key details are summarized below.

We recruited two two high schools in the U.S.A. and four high schools in Shanghai to participate. In each school, student subjects take a 25 minute, 25 question mathematics test which is made up of multiple choice and free answer questions that were given on past editions of the Programme for International Student Assessment (PISA). Students are randomized into either treatment or control. Members of the treatment group are given a financial incentive that is based on their test performance. Right before taking the test, they are given an envelope with $25 in cash (or the equivalent in RMB in Shanghai), and are told that $1 will be taken away for each question that is answered incorrectly. The students had no advance notice of the task they would be doing or of the financial incentives. The control group takes the test with no financial incentives.

The exam is administered by computer so scores are available immediately after the test is completed.

The main experiment was conducted in 2016 at two high schools in the U.S. and at three schools in Shanghai.
U.S. School 1 is a high performing private boarding school.
U.S. School 2 is a low performing public school.
At both of these schools, all 10th grade students were required to participate.

Shanghai schools 1 through 3 include one below-average performing school, one school with performance that is just above average, and one
school with performance that is well above average. Two classes each of 10th grade math students at schools 1 and 2 and four classes of 10th grade math students at school 3 were randomly selected to participate.

in 2018, we reran the experiment in Shanghai at schools 2 and 3 and at a new school, school 4, whose performance is also well above average.

Logistics required different randomization procedures of students into treatment or control (described in more detail below):
U.S. School 1: students were randomized into treatment or control at the individual level.
U.S. School 2: students were randomized into treatment or control at the class level.
Shanghai schools 1-3 in 2016 were randomized into treatment or control at the class level.
Shanghai schools 2-4 in 2018 were randomized into treatment or control at the individual level.

Design details

Randomization method
The randomization is stratified by school. Logistics required different randomization procedures of students into treatment or control. We randomized at the class level in the lower performing school (school 2) in the U.S. and in the 2016 sessions in Shanghai. We randomized at the individual level in the higher performing school in the U.S. and in the 2018 sessions in Shanghai. In the U.S., we stratified by school and re-randomized to achieve balance on the following baseline characteristics: gender, ethnicity and mathematics class level/track: low, regular, and honors. For each school's randomization, we re-randomized until the p-values of all tests of differences between Treatment and Control were above 0.4. In the 2016 Shanghai sessions, we stratified the randomization by school (baseline demographics were not available at the time of randomization). In the 2018 Shanghai sessions, we stratified the randomization by class, gender, and senior entrance exam score quartile.
Randomization unit
The randomization is stratified by school. Within each school the level of randomization varied because of logistical constraints. U.S. School 1: students were randomized into treatment or control at the individual level. U.S. School 2: students were randomized into treatment or control at the class level. At Shanghai schools 1-3 in 2016, students were randomized into treatment or control at the class level. At Shanghai schools 2-4 in 2018, students were randomized into treatment or control at the individual level.
Clusters
U.S.: 131 clusters. In U.S. school 2, the randomization was done at the individual level so the size of each cluster is 1 for that school. Shanghai: 384 clusters. In the 2018 sessions, the randomization was done at the individual level so the size of each cluster is 1 for those sessions.
Observations
The U.S. sample includes 447 students (227 in control and 220 in treatment). The Shanghai sample includes 656 students (333 in control and 323 in treatment).
Arms
U.S.: 227 students organized into 64 clusters (12 of size n>1, 52 of size n=1) in control. 220 students organized into 67 clusters (13 of size n>1, 54 of size n=1) in treatment. Shanghai: 333 students organized into 196 clusters (4 of size n>1, 190 of size n=1) in control 323 students organized into 188 clusters (4 of size n>1, 184 of size n=1) in treatment
Primary outcomes
The study examines four outcome variables, all related to performance on the test: Student level outcomes (N = 447 in U.S., N = 656 in Shanghai) 1. number of questions answered correctly (out of 25), standardized by subtracting the sample mean and dividing by the sample standard deviation. Question response level outcomes (N = 447 students x 25 questions = 11,175 observations in U.S., N = 656 students x 25 questions = 16,400 observations in Shanghai) 2. The probability that question i is attempted by student j: a) over all 25 questions b) over questions 1-13 c) over questions 14-25 3. The probability that student j answered question i correctly (sample of attempted questions only) a) over all 25 questions b) over questions 1-13 c) over questions 14-25 4. The probability that student j answered question i correctly (all questions, both attempted and not attempted) a) over all 25 questions b) over questions 1-13 c) over questions 14-25

Papers from this trial

  1. Uri Gneezy, John A. List, Jeffrey A. Livingston, Sally Sadoff, Xiangdong Qin, Yang Xu (2017) "Measuring Success in Education: The Role of Effort on the Test Itself." NBER Working Paper No. 24004 https://www.nber.org/papers/w24004

Cited by

No citation data available for this registration yet.

Keywords