Nimble evaluations of an online entrepreneurship and STEM training program in schools

How to cite

McKenzie, David; Asanov, Igor. 2018. "Nimble evaluations of an online entrepreneurship and STEM training program in schools." AEA RCT Registry. AEARCTR-0003553. https://doi.org/10.1257/rct.3553-3.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

David McKenzie* , Igor Asanov

* Primary investigator

Registry ID
AEARCTR-0003553
DOI
https://doi.org/10.1257/rct.3553-3.0
First registered
Last updated
Status
Completed
Trial dates
2019-02-01 – 2020-12-01
Countries
Ecuador ()
JEL codes
I21, L26, O15
Registry record
http://www.socialscienceregistry.org/trials/3553

Abstract

Opportunity-focused, high-growth entrepreneurship and science-led innovation are crucial for continued economic growth and productivity. Working in these fields offers the opportunity for rewarding and high-paying careers. However, the majority of youth in developing countries do not consider either as job options, affecting their choices of what to study. Youth may not select these educational and career paths due to lack of knowledge, lack of appropriate skills, and lack of role models. We are working to provide a scalable approach to overcoming these constraints through an online education course for secondary school students that covers entrepreneurial soft skills, scientific methods, and interviews with role models. This course will be taken by students during class time, under teacher supervision.
There are then three policy problems that we aim to use nimble evaluations to provide evidence on:
1) Maximizing take-up: How can the Ministry of Education maximize take-up of the program in selected schools, ensuring that teachers use it in their classes?
2) Content ordering: Is it better to try to inspire students first through interviews with role models, or to first build their knowledge and skills so that they connect better with role models?
3) Standardized or Adaptive Learning: Should the same assessment tools be used for all students, or are learning gains higher with exercises that adapt to the interests and abilities of students?

Intervention

The online course will be taught in schools over 12 weeks in 2019, and will contain the following content: (1) entrepreneurship-related soft skills – this will cover personal initiative and negotiation skills; (2) basic scientific methods – this will cover ideas of the experimental method, and basic statistics; and (3) role model interviews with entrepreneurs and scientists. Placebo courses will be offered to a control group for measuring the overall effectiveness of these courses.

The following nimble evaluation interventions will be undertaken to improve implementation fidelity and effectiveness of these courses:
1) Interventions to maximize teacher take-up: the Ministry of Education plans to train the teachers online about the new course content. We will test whether in-person training is cost-effective in delivering better compliance and usage. We will also test whether providing benchmarking information on take-up in other schools leads to higher usage.
2) Varying content ordering: we will randomize the ordering of modules to learn whether it is better to introduce students to role models before or after they get the skill content.
3) Introducing adaptive-learning exercises: the standard course will have the same content and exercises for all students. We will test whether student learning gains are higher if exercises are adapted to student skills and interests.

Experimental design

The three nimble evaluations will have randomization at different levels:
1) Teacher take-up: we will cross-randomize two treatments at the school level: (i) whether teachers are trained online about the new program, or in-person; and (ii) whether teachers receive weekly benchmarking information on take-up and usage in other schools.
2) Content-ordering: classes in the treatment schools will be randomized at the class level as to whether they receive the role model content at the start of the intervention or at the end.
3) Adaptive or non-adaptive exercises: individuals in the treated schools will be randomized at the individual level as to whether they receive standard exercises which are the same for everyone, or to whether they get adaptive-exercises which vary depending on initial performance and student interests.

Design details

Randomization method
Randomization to be done in office by computer
Randomization unit
There are three different levels of randomization: - The take-up interventions will be randomized at the school-level. - The content ordering interventions will be randomized at the class-level - The adaptive or non-adaptive exercise interventions will be randomized at the individual level
Clusters
We plan to work with 110 schools. These will be randomized into 70 treatment schools and 40 control schools. Across these 110 schools we will carry out a 2x2 randomization into whether training is in person or not, and whether benchmarking is provided or not. Within the 70 treatment schools, there will be approximately 350 classes. We will randomize the ordering of topics in these classes. The adaptive interventions are not clustered, but randomized at the individual level.
Observations
We anticipate 110 schools, and will work with levels 10 and 11 in these schools, for 220 grades. There are approximately 5 classes per school, so approximately 550 classes in total, and approximately 180 students per school, for aproximately 20,000 students in total.
Arms
Intervention 1: 55 schools in-person training, 55 schools online training 55 schools with benchmarking, 55 schools no benchmarking Intervention 2: approximately 175 classes with role model content at the start of the course, approximately 175 classes with role model at the end Intervention 3: approximately 10,000 students with adaptive exercises, 10,000 with non-adaptive.
Primary outcomes
Active take-up: Hours per student actively spent on the platform Career attitudes: interest in entrepreneurial and scientific careers Learning: Students’ performance on subject-specific tests Steps towards career: Action based outcomes.

Materials

Papers from this trial

  1. Igor Asanov, Anastasiya-Mariya Asanov, Thomas Astebro, Guido Buenstorf, Bruno Crepon, Francisco Pablo Flores T, David McKenzie, Mona Mensmann and Mathis Schulte (2023) "System-, teacher-, and student-level interventions for improving participation in online learning at scale in high schools", PNAS 120(3): e2216686120, July 17 https://www.pnas.org/doi/10.1073/pnas.2216686120

Cited by

No citation data available for this registration yet.

Keywords