Estimating the Educational Production Function in EdTech: Experimental Evidence from Russia

How to cite

Loyalka, Prashant; Fairlie, Robert. 2019. "Estimating the Educational Production Function in EdTech: Experimental Evidence from Russia." AEA RCT Registry. AEARCTR-0004126. https://doi.org/10.1257/rct.4126-1.3000000000000003

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Prashant Loyalka* , Robert Fairlie

* Primary investigator

Registry ID
AEARCTR-0004126
DOI
https://doi.org/10.1257/rct.4126-1.3000000000000003
First registered
Last updated
Status
On going
Trial dates
2018-10-01 – 2022-12-31
Countries
Private
JEL codes
C93, I21, J24, O15
Registry record
http://www.socialscienceregistry.org/trials/4126

Abstract

Although EdTech is rapidly expanding in developing countries little is known about its marginal productivity. Based on a large field experiment we will provide the first test in the literature of whether the educational production function is concave, convex or linear in computer assisted learning (CAL). Results are fundamental to understanding optimal investments in technology-based education. We will use two treatment arms with different levels of time on CAL and a pure control group to estimate effects on student achievement. We will also examine cost-effectiveness and whether different types of students face different production functions. Using two treatment intensities will also allow us to avoid the potential problem of choosing a treatment intensity that is too low or too high, and thus might provide a misleading evaluation of the effectiveness of CAL. Theory and the newness of EdTech in developing countries provide almost no guidance on optimal levels of investment.

Intervention

A large online technology company in Russia (hereafter “the provider”), has developed an online education platform through which students are given math and language arts items to solve. The platform has more than 10 thousand items across various math and language sub-content areas for grades 2 to 4. The items are aligned with national educational standards and curricula for primary schools.

In both the CAL Dosage X and CAL Dosage 2X treatment arms, teachers are asked to assign the online items for homework. The dosages were chosen based on a large number of interviews that the provider conducted with teachers outside of the study sample and prior to the experiment. Dosage X is 25 minutes (10 online items) per subject per week. Dosage 2X is 50 minutes (20 online items) per subject per week.

Experimental design

The RCT involves more than 6,000 third grade schoolchildren in 343 classes/schools in two provinces of Russia. The RCT includes three treatment arms: an “X” dosage CAL arm in which students have approximately 25 minutes per week of math CAL and 25 minutes of Russian language CAL, a “2X” dosage CAL arm in which students have approximately 50 minutes of math CAL and 50 minutes of Russian language CAL, and a control arm. The number of schools/classes in each treatment arm is as follows:

A. CAL Dosage X (T1) 115 classes (in 115 schools)
B. CAL Dosage 2X (T2) 113 classes (in 113 schools)
C. Control (C) 115 classes (in 115 schools)

Design details

Randomization method
Randomization done by a computer.
Randomization unit
Sample strata or blocks were created by placing the 6 classes with the closest mean grade 3 math scores in a region in a strata. Altogether, this resulted in 56 strata. Classes were then randomly allocated within these strata to one of three different treatment conditions (T1 = CAL Dosage X, T2 = CAL Dosage 2X, or C = Control or No CAL): A. CAL Dosage X (T1) 115 classes (in 115 schools) B. CAL Dosage 2X (T2) 113 classes (in 113 schools) C. Control (C) 115 classes (in 115 schools)
Clusters
343 schools/classes
Observations
~6000 students
Arms
A. CAL Dosage X (T1) 115 classes (in 115 schools) B. CAL Dosage 2X (T2) 113 classes (in 113 schools) C. Control (C) 115 classes (in 115 schools)
Primary outcomes
The primary outcome variables for the trial will be student math and language achievement at the end of the school year (as measured by the IPIPS+ exam). In the analyses, we will convert the math and language exam scores from the endline into z-scores (subtracting each students’ endline subject-specific score by the average endline subject-specific score of the control sample and dividing the standard deviation of the endline subject-specific score of the control sample). Secondary outcome variables will include student grades in math and language (which we will collect directly from the schools and teachers), the degree to which students like math and language subjects (using a subjective scale), and student and teacher reports of time spent on different subject-specific study activities.

Cited by

No citation data available for this registration yet.

Keywords