Nimble evaluations of an online entrepreneurship and STEM training program in schools
How to cite
McKenzie, David; Asanov, Igor. 2018. "Nimble evaluations of an online entrepreneurship and STEM training program in schools." AEA RCT Registry. AEARCTR-0003553. https://doi.org/10.1257/rct.3553-3.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Opportunity-focused, high-growth entrepreneurship and science-led innovation are crucial for continued economic growth and productivity. Working in these fields offers the opportunity for rewarding and high-paying careers. However, the majority of youth in developing countries do not consider either as job options, affecting their choices of what to study. Youth may not select these educational and career paths due to lack of knowledge, lack of appropriate skills, and lack of role models. We are working to provide a scalable approach to overcoming these constraints through an online education course for secondary school students that covers entrepreneurial soft skills, scientific methods, and interviews with role models. This course will be taken by students during class time, under teacher supervision.
There are then three policy problems that we aim to use nimble evaluations to provide evidence on:
1) Maximizing take-up: How can the Ministry of Education maximize take-up of the program in selected schools, ensuring that teachers use it in their classes?
2) Content ordering: Is it better to try to inspire students first through interviews with role models, or to first build their knowledge and skills so that they connect better with role models?
3) Standardized or Adaptive Learning: Should the same assessment tools be used for all students, or are learning gains higher with exercises that adapt to the interests and abilities of students?
Intervention
The online course will be taught in schools over 12 weeks in 2019, and will contain the following content: (1) entrepreneurship-related soft skills – this will cover personal initiative and negotiation skills; (2) basic scientific methods – this will cover ideas of the experimental method, and basic statistics; and (3) role model interviews with entrepreneurs and scientists. Placebo courses will be offered to a control group for measuring the overall effectiveness of these courses.
The following nimble evaluation interventions will be undertaken to improve implementation fidelity and effectiveness of these courses:
1) Interventions to maximize teacher take-up: the Ministry of Education plans to train the teachers online about the new course content. We will test whether in-person training is cost-effective in delivering better compliance and usage. We will also test whether providing benchmarking information on take-up in other schools leads to higher usage.
2) Varying content ordering: we will randomize the ordering of modules to learn whether it is better to introduce students to role models before or after they get the skill content.
3) Introducing adaptive-learning exercises: the standard course will have the same content and exercises for all students. We will test whether student learning gains are higher if exercises are adapted to student skills and interests.
Experimental design
The three nimble evaluations will have randomization at different levels:
1) Teacher take-up: we will cross-randomize two treatments at the school level: (i) whether teachers are trained online about the new program, or in-person; and (ii) whether teachers receive weekly benchmarking information on take-up and usage in other schools.
2) Content-ordering: classes in the treatment schools will be randomized at the class level as to whether they receive the role model content at the start of the intervention or at the end.
3) Adaptive or non-adaptive exercises: individuals in the treated schools will be randomized at the individual level as to whether they receive standard exercises which are the same for everyone, or to whether they get adaptive-exercises which vary depending on initial performance and student interests.
Design details
Materials
Papers from this trial
- Igor Asanov, Anastasiya-Mariya Asanov, Thomas Astebro, Guido Buenstorf, Bruno Crepon, Francisco Pablo Flores T, David McKenzie, Mona Mensmann and Mathis Schulte (2023) "System-, teacher-, and student-level interventions for improving participation in online learning at scale in high schools", PNAS 120(3): e2216686120, July 17 https://www.pnas.org/doi/10.1073/pnas.2216686120
Cited by
No citation data available for this registration yet.