Do Experiments Teach Basic Economics Lessons Better than Traditional Teaching Techniques?

How to cite

Livingston, Jeffrey; Gelfer, Sacha; Roy, Sutanuka. 2021. "Do Experiments Teach Basic Economics Lessons Better than Traditional Teaching Techniques?." AEA RCT Registry. AEARCTR-0008744. https://doi.org/10.1257/rct.8744-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jeffrey Livingston* , Sacha Gelfer , Sutanuka Roy

* Primary investigator

Registry ID
AEARCTR-0008744
DOI
https://doi.org/10.1257/rct.8744-1.0
First registered
Last updated
Status
Completed
Trial dates
2021-05-17 – 2021-12-16
Countries
United States of America (Massachusetts)
JEL codes
A22, C93
Registry record
http://www.socialscienceregistry.org/trials/8744

Abstract

This study examines whether in-class experiments help students learn important Economics lessons better than traditional methods. To address this question, we conduct an experiment in undergraduate Introductory Microeconomics and Macroeconomics classes. In each section of a class, we randomize two lessons out of four potential candidates into treatment or control. Treatment lessons are taught using an in-class experiment or similar active-learning activity that is designed to illustrate the topic. Control lessons are taught using traditional techniques. Our design follows the recommendations of Wozny et al. (2018) by randomizing across lessons within a given course, increasing the power of our statistical tests. For each of these four lessons, the instructors give a multiple choice quiz after they have finished teaching the topic. We also ask several multiple choice questions on the course final exam related to each of these lessons. This allows us to measure both whether the experiments help students learn key lessons better than traditional techniques in both the short term and the long term. We also investigate whether the effect of these techniques varies by gender, topic area (micro vs. macro) and instructor type (adjunct, lecturer, or tenured/tenure track). The classes in our sample are taught mostly by instructors who are inexperienced in the use of experiments as a teaching tool, so our results should be generalizable to schools that have not commonly used this technique.

Intervention

We chose four topic areas that are covered in each participating section of the course to be part of the experiment. In the microeconomics sections, these topics include:

1. Production possibilities and gains from trade
2. The impact of supply and demand shifts on equilibrium
3. The impact of price ceilings and price floors on a market
4. The decisions of firms under perfect competition in the short run and long run

In the macroeconomics sections, these topics include:
1. Production possibilities and gains from trade
2. Currency speculation
3. Aggregate demand and aggregate supply, and monetary policy
4. Labor and goods general equilibrium
The experimental design is based on Wozny, Balser, and Ives (2018). In this paper, the authors recommend randomizing treatment at the lesson level within classes. Our identification strategy follows their recommendation. We block by section and randomize which two of these four topics are assigned to treatment within each section.

For treatment topics, instructors are asked to follow this procedure:

1. At the beginning of teaching the topic, before teaching any of the material, conduct the related in-class active learning activity.
2. Teach the related material as the instructor normally would, making efforts to refer back to specific things that happened or that students experienced during the active learning material when possible and appropriate.
3. After finishing covering the topic area, give the related five question quiz at the beginning of the next class with a ten minute time limit.
4. At the conclusion of the class, give 12 more multiple choice questions, 3 on each topic, as part of the final exam.

Experimental design

The experiment is being conducted with 10 sections of Principles of Microeconomics and 5 sections of Principles of Macroeconomics. In each section, we randomize two of four topics to be partially taught using a particular in-class experiment or similar active-learning exercise.

In the microeconomics classes, these topics are:
1. Production possibilities and gains from trade
2. The impact of supply and demand shifts on equilibrium
3. The impact of price ceilings and price floors on a market
4. The decisions of firms under perfect competition in the short run and long run

In the macroeconomics classes, these topics are:
1. Production possibilities and gains from trade
2. Currency speculation
3. Aggregate demand and aggregate supply, and monetary policy
4. Labor and goods general equilibrium

For the two topics that are randomized into treatment, instructors are trained in how to administer the experiment. They are asked to follow this procedure:

1. At the beginning of teaching the topic, before teaching any of the material, conduct the related in-class active learning activity.
2. Teach the related material as the instructor normally would, making efforts to refer back to specific things that happened or that students experienced during the active learning material when possible and appropriate.
3. After finishing covering the topic area, give the related five question quiz at the beginning of the next class with a ten minute time limit.
4. At the conclusion of the class, give 12 more multiple choice questions, 3 on each topic, as part of the final exam.

For control topics, the 5 question quiz is also given at the beginning of the first class after the instructor has completed teaching of that topic.

Thus, we employ a within-subject design. Each student generates two treatment observations and two control observations. Identification is based on this within-student variation.

Design details

Randomization method
Randomization was blocked on class section, then done via random number generator in Excel. For each participating section, each topic is assigned a random number. The topics receiving the two highest random numbers were assigned to treatment.
Randomization unit
The unit of randomization is topic/lesson within each class.
Clusters
10 sections of microeconomics and 5 sections of macroeconomics.
Observations
Average class sizes are around 25. We therefore expect a total of 15x25x4 = 1500 observations.
Arms
By design, the number of observations in each treatment arm (only two - treatment and control) will be about 750. Each student participates in four lessons, two of which are taught by experiment and two of which are taught normally. This of course assumes perfect attendance, participation in all quizzes, etc.
Primary outcomes
1. scores on 5 question multiple choice quizzes covering a course topic that are given immediately after the teaching of that topic has been completed (a measure of short-term learning) 2. scores on 12 multiple choice final exam questions covering the topics that are part of our experiment (a measure of longer-term learning) We will examine how our treatment impacts: 1. raw score on these quizzes/tests 2. score on these tests standardized by topic (constructed by subtracting the topic within-sample mean score and dividing by the topic within-sample standard deviation; means and standard deviations calculated using control observations only) We plan on estimating: 1. the overall treatment effect of in-class experiments on student learning (as measured by quiz and final exam question scores) 2. the extent to which this treatment effect varies: a) by gender b) by topic area (micro vs. macro) c) by instructor type (adjunct, lecturer, or tenured/tenure track) as well as subgroups based on each of these categories, e.g. by gender within each subject area.

Cited by

No citation data available for this registration yet.

Keywords