Solving the 2 Sigma Problem with Khan Academy: A Pilot Study

How to cite

Oreopoulos, Philip; Price, Joseph. 2020. "Solving the 2 Sigma Problem with Khan Academy: A Pilot Study." AEA RCT Registry. AEARCTR-0006167. https://doi.org/10.1257/rct.6167-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Philip Oreopoulos* , Joseph Price

* Primary investigator

Registry ID
AEARCTR-0006167
DOI
https://doi.org/10.1257/rct.6167-1.0
First registered
Last updated
Status
In development
Trial dates
2020-08-17 – 2021-08-24
Countries
Private
JEL codes
I2
Registry record
http://www.socialscienceregistry.org/trials/6167

Abstract

Computer Assisted Learning (CAL) is educational software designed to help students progress through material at their own pace while receiving feedback and advice, similar to the kind that a tutor might provide but on a computer. CAL also makes it easier for teachers to monitor and grade progress. Teachers can assign new topics based on each students’ level of understanding. Despite a large and increasing body of convincing research suggesting large benefits of adopting CAL, many administrators and teachers still shy away. Uncertainty about how to use the software, concerns about subscription costs, and lack of support may explain this reluctance. This pilot aims to develop and test a program that simplifies the adoption of CAL and helps ensure its effective use. The pilot uses Khan Academy (KA), one of the most popular CAL programs, as a tool for Grades 3-8 math teachers to use in class and as graded homework. The program has the potential to save time for teachers, lower costs, and offer a more customizable, enjoyable, and effective learning experience for students. The goal of the pilot is to work out design details for maximizing engagement and simplifying usage. The overall goal is to provide convincing evidence for policy makers, administrators, and teachers to want to adopt the program on a state-wide or national scale.

Intervention

We propose to pilot a program that provides training and support for teachers to use KA effectively in Grades 3-8 mathematics. We chose mathematics because previous research (cited above) finds CAL can be particularly effective with this subject. We chose Grades 3-8 because KA’s resources for these levels are particularly well developed. The pilot program has the potential to save time for teachers and offer a free, more customizable, enjoyable, and effective learning experience for students. The pilot would help determine best practices for the implementation of a larger randomized controlled trial. Qualitative and quantitative analyses would be used to identify key implementation challenges and solutions, whether classes are provided online or in-person. If a sufficiently large number of classes participate, we will also implement the study as its own randomized controlled trial.

Experimental design

Randomized at the classroom or grade level. Treatment group receives program in the Fall, 2020, control group receives it in the Spring, 2021.

Design details

Randomization method
Randomization done in office by a computer (using STATA).
Randomization unit
Class or grade-level.
Clusters
10 schools, 10 classes each, for a total of 100 classes.
Observations
Assuming 20 students per class, target sample size is about 2,000 students.
Arms
50 classes treated, 50 contol.
Primary outcomes
Intermediate outcomes will come from class attendance and grade records. We will also use usage data from Khan Academy. A support letter from them is attached. We will also match students to their standardized math achievement scores from the Utah Performance Assessment System (U-Pass). Qualitative data would be collected on student/teacher/parent experiences and feedback from administrators about the program.

Cited by

No citation data available for this registration yet.

Keywords