Nudging Student Effort to Accurately Measure Student Ability: Maximizing the Value of Assessment Data through Nimble Interventions

How to cite

Baron, Juan; Adelman, Melissa. 2018. "Nudging Student Effort to Accurately Measure Student Ability: Maximizing the Value of Assessment Data through Nimble Interventions." AEA RCT Registry. AEARCTR-0003415. https://doi.org/10.1257/rct.3415-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Juan Baron* , Melissa Adelman

* Primary investigator

Registry ID
AEARCTR-0003415
DOI
https://doi.org/10.1257/rct.3415-1.0
First registered
Last updated
Status
In development
Trial dates
2019-01-14 – 2019-05-30
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/3415

Abstract

Low-stakes, diagnostic student learning assessments are an important tool for countries in their response to the global “learning crisis.” Performance on such assessments reflects a combination of student learning and effort exerted. Therefore, test scores from such assessments may overestimate the true magnitude of learning gaps when differences in effort exist, and such effort differences may affect how assessment results should be interpreted (for example, the meaning of test score differentials across socioeconomic groups). Several randomized evaluations, primarily in the United States, have demonstrated that external incentives (monetary and non-monetary) and framing can significantly affect students’ performance on otherwise low-stakes tests (Braun, Kirsch, and Yamamoto 2011; Levitt et al. 2016; Gneezy et al. 2017). As countries advance in their reform of student assessment systems and continue to replace high-stakes assessments with low-stakes ones, the issue of whether low-stake assessments are capturing the reality of students’ knowledge and competences becomes more and more relevant, and even more so, if some groups of students (i.e. those from disadvantage backgrounds) tend to exert more or less effort than others.

The Dominican Republic (DR), which came in last both in PISA 2015 and in the TERCE regional assessment in 2013, has embarked on an ambitious assessment agenda that includes diagnostic learning assessments of 3rd, 6th, and 9th grades and their potential use for targeted in-service teacher training and other school-support activities. The results of these assessments will therefore inform the allocation of scarce resources, and DR Ministry of Education (MINERD) policymakers would like to maximize the extent to which they accurately measure learning. Relatedly, MINERD policymakers are interested in identifying low-cost, scalable approaches to increasing student effort in day-to-day, low-stakes tasks, where effort exerted can have a cumulatively important effect on how much students learn. This is particularly critical for poor and disadvantaged students, who are less likely to have direct experiences or role models of high effort leading to high achievement, and are also less likely to have parents who provide incentives for school achievement (Gottfried, Fleming, and Gottfried 1998; Austen-Smith and Fryer 2005).

Experimental design

The sample of schools for the study will be drawn randomly from the universe of public secondary schools in the Dominican Republic, stratified by district poverty rate. These school will be randomly assigned to 1 of 3 treatment groups, as such:
Treatment 1: Random sample of public secondary schools – will receive a visit from local MINERD staff to announce and carry out the awarding of external incentive (same day as assessment). Visit will be pre-announced to school director and teachers, without specifics on the purpose.
Treatment 2: Random sample of public secondary schools – will receive a packet addressed to school director and 9th grade teacher(s) with Growth Mindset exercise materials, instructions to conduct the exercise a few months prior to the assessment, and instructions requesting that a photo be texted or emailed to the Ministry showing the exercise in progress.
Treatment 3: Random sample of public secondary schools – will receive a packet addressed to school director and 9th grade teacher(s) with Self-Affirmation exercise materials, instructions to conduct the exercise a few months prior to the assessment, and instructions requesting that a photo be texted or emailed to the Ministry showing the exercise in progress.

A fourth group drawn in the same way will constitute the control group.

Design details

Randomization method
Randomization done in office by a computer
Randomization unit
School
Clusters
600 to 800 school in total
Observations
18,000 to 24,000 students in ninth grade
Arms
Treatment 1: 150 - 200 schools Treatment 2: 150 – 200 schools Treatment 3: 150 – 200 schools Comparison: 150 - 200 schools
Primary outcomes
Students' 9th grade assessment score in the national diagnostic test

Cited by

No citation data available for this registration yet.

Keywords