Hiring Effective Math Teachers: More Knowledge vs. Better Delivery
How to cite
Kim, Seonghoon; Kim, Hyuncheol. 2016. "Hiring Effective Math Teachers: More Knowledge vs. Better Delivery." AEA RCT Registry. AEARCTR-0001687. https://doi.org/10.1257/rct.1687-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Teacher quality is one of the key determinants of students’ academic success. We examine the impact of teacher screening criteria on academic outcomes of primary school students to better hire effective teachers. Specifically, we consider two different traits of a teacher that can potentially improve academic outcomes: 1) a teacher candidate’s math test score and 2) his/her performance on a teaching demonstration. Our study takes place in the context of the after school teacher recruitment drive in rural Malawi by our collaborating NGO, Africa Future Foundation (AFF). During the pre-employment testing session, AFF conducted the math test and teaching demonstration for 347 teacher applicants. Then, AFF randomly divided them into two groups: “math test” and “teaching demonstration” group. 132 out of 173 in the math test group were hired based on their math test score as after school teachers. 130 out of 174 in the teaching demonstration group were hired based on their teaching demonstration evaluation. Hired teachers were randomly assigned to 414 classes in 31 schools, with 122 classes comprising of 4-7th graders. A class size of 4, 6, 8, or 10 was randomly determined for each tutoring class. We compare the performance of teachers (students’ test score) selected via two different channels over one year (three semesters) to shed light on effective teacher screening.
Intervention
We hire after school math teachers randomly based on either their math test score or teaching demonstration and measure the impacts on students’ test scores. Below are the details of the experiment.
Africa Future Foundation (AFF) recruited after school math teachers to teach 4th to 7th graders at the onset of the after school tutoring program in the catchment district of rural Malawi (T.A. Chimutu). (As of September, 2016, students are now 5th to 8th graders.) The total number of applicants was 347. Job applicants participated in a survey which collects information on demographics, educational background, and non-cognitive traits. They also took a math test and performed a teaching demonstration which AFF used as evaluation criteria for recruitment.
Then, AFF randomly divided the applicants into two groups: 173 were assigned to the “math test” group, while the other 174 were assigned to the “teaching demonstration” group. 132 out of 173 in the math test group were hired based on the math test score as after school teachers. 130 out of 174 in the teaching demonstration group were hired based on their teaching demonstration evaluation.
After the one-week-long training, after school teachers were randomly assigned to students from 122 grades in 31 schools. In addition, AFF randomly selected two thirds of the grade-schools (for example, 80 grade-schools in the first semester) for the after school tutoring treatment. In the selected grades, a randomly selected one half of the students is eligible for after school tutoring. In this manner, everyone will be eligible at least once during the school calendar year, which consists of three semesters. In addition, students eligible for tutoring are randomly assigned to groups of tutoring class sizes of 4, 6, 8 or 10.
Experimental design
1. Job advertisements for after school tutors were posted throughout the schools and villages in the catchment area beginning from January 25th, 2015. The NGO advertised vacancies through posters which describe the job and work conditions. High school graduation was the minimum requirement for the position, and applicants were asked to submit an application form along with their Malawi School Leaving Exam transcript (secondary school diploma). Applications were received at 31 primary schools in the catchment area and AFF office. AFF invited applicants to participate in the recruitment meeting in which they participated in the baseline survey, took a math test, and performed a teaching demonstration.
2. The math test consists of 90 questions, which are chosen from 4th to 12th grade textbooks. For the teaching demonstration, applicants were given 2 math problems from the secondary school level (9-10th grade level). Each applicant had 10 minutes to prepare the demonstration and 10 minutes to present it. Several pairs of trained AFF staff members were randomly assigned to the applicants in order to evaluate each applicant’s teaching skill. Specifically, they evaluated overall performance based on the applicant’s engagement with the audience, delivery of mathematical elements in the problems, time management, and enthusiasm.
3. AFF randomly divides applicants into two groups. 173 were assigned to the “math test” group, while the other 174 were assigned to the “teaching demonstration” group. 132 out of 173 in the math test group were hired based on their math test score as after school teachers. 130 out of 174 in the teaching demonstration group were hired based on their teaching demonstration evaluation.
4. Selected after school teachers were required to complete a one-week-long training, during which they were subjectively evaluated by evaluators (regular employees of AFF). Trainees participated in a mock teaching session, where they presented how to solve a pre-assigned math problem from an assigned math course. Evaluators then scored trainees on a scale from 1 to 3 in terms of engagement with the audience, delivery of mathematical elements in the problems, time management, and enthusiasm.
5. The first semester of the experiment is the third semester of the academic year 2015/16 (April 11, 2015 - July 15, 2016). 122 grades (4th-7th grade) in 31 schools were randomized to either tutoring or no tutoring based on students’ grade and average class level computed from students’ math exam score.
Group 1 (42 grades): no tutoring (5,303 students)
Group 2 (78 grades): tutoring (6,396 students). Out of 6,396 students, 2,849 students were randomly assigned and 3,547 students were not assigned to tutoring.
6. As a result of the above process, 2,849 students out of 6,396 students were randomly assigned to after school tutoring in the third semester of the academic year 2015/2016. The remaining students will be eligible for the tutoring in the following semesters.
7. Each teacher was also randomly assigned to 2 out of 414 after school classes. Each class was held twice a week, either Monday-Wednesday or Tuesday-Thursday. The duration of the tutoring program was 7 weeks per semester, and eligible students took 13 classes in total.
8. There are four tutoring class sizes: 4, 6, 8, and 10. These four class sizes were randomly distributed within each school-grade. For example, when one school-grade consists of 35 students in the tutoring treatment, each of the four class sizes (4, 6, 8, and 10) is initially assigned in a random order.
9. Students were then assigned to the classes in descending order of their baseline math scores. As a result, they were in the same class as those with a similar baseline math score.
10. The collaborating NGO commits to hire the after school teachers for at least three semesters. In the second semester of the experiment (i.e. the 1st semester of the academic year 2016/17), 120 grades (5th-8th grade; students moved on to the next grade as the school year changed) in 31 schools were randomized to either tutoring or no tutoring. School-grades which were not in the tutoring group (42 grades) in the first semester of tutoring were automatically included in the tutoring group. Of the school-grades which received tutoring in the first semester (78 grades), 50% (38 grades) were randomly selected for the tutoring group, with the remaining 40 grades assigned to the no tutoring group. The random assignment of tutors, class sizes, and students was conducted in the same manner as that of the first semester. We will continue to provide the tutoring service to students through the third semester of the experiment (the 2nd semester of the academic year 2016/17).
Design details
Cited by
No citation data available for this registration yet.