A Field Experiment on Adaptive Learning Applying a Machine Learning Algorithm

How to cite

Zipperle, Isabell; Klausmann, Tim; Köppel, Marius; Schunk, Daniel. 2021. "A Field Experiment on Adaptive Learning Applying a Machine Learning Algorithm." AEA RCT Registry. AEARCTR-0007637. https://doi.org/10.1257/rct.7637-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Isabell Zipperle* , Tim Klausmann , Marius Köppel , Daniel Schunk

* Primary investigator

Registry ID
AEARCTR-0007637
DOI
https://doi.org/10.1257/rct.7637-1.0
First registered
Last updated
Status
In development
Trial dates
2018-01-22 – 2022-12-31
Countries
Germany ()
JEL codes
C69, C93, I20, I21
Registry record
http://www.socialscienceregistry.org/trials/7637

Abstract

It remains an open question how to adapt and individualize learning contents. To tackle this in a digital learning context, we developed an algorithm based on a convolutional neural network that assigns tasks to the learners. Our application is a large online learning platform in which we run a randomized controlled trial. Participants are randomized into three groups: two treatment groups – a group-based adaptive treatment group and an individualized adaptive treatment group – and one control group. We analyze the difference between the three groups with respect to effort learners provide within the platform, their performance within the platform, formative assessments within the platform and their final high-stake standardized exams (summative assessment).

Intervention

There is a control group, a group-based adaptive treatment group, and an individualized adaptive treatment group. The control group receives tasks randomly drawn from the pool of tasks available in the learning platform. Both treatment groups are affected by our “intervention”. Tasks are assigned to learners based on pre-tested distributions that optimize task difficulty and learning effort, which is the number of future tasks that learners will complete in the platform. A machine learning algorithm predicts task difficulty and future effort. The groups differ in the degree of individualization of assigned tasks. The group-based adaptive treatment group receives tasks that are adapted to the mean learner in the platform. The individualized adaptive treatment group receives tasks that are adapted to each individual learner.

Experimental design

The setting of our experiment is a college-level online learning platform that students use outside the classroom (for more information of the platform see Sponsors & Partner section). Learning in the platform is unmonitored by their teachers. Originally, the platform consists of learning videos and tasks that are matched to the videos and that follow the purpose of deepening the understanding of the video contents. We added a section to the platform that combines learning materials to quizzes of ten tasks. Our intervention algorithm assigns tasks to the learners whenever they choose to take a quiz.

Design details

Randomization method
Computer-based randomization by script on a server of the cooperation partner
Randomization unit
Individual
Clusters
10,000
Observations
10,000
Arms
3,300 group-based adaptive treatment group 3,300 individualized adaptive treatment group 3,300 control group
Primary outcomes
Number of attempted tasks
Secondary outcomes
Test score in formative assessments (mock exams) on the platform Test score in summative assessments (standardized high-stake final exams)

Cited by

No citation data available for this registration yet.

Keywords