Evaluating a Whatsapp based Self Development Program to Enhance English Language Skills and Teacher Agency in Pakistan
How to cite
Liaqat, Asad; Duhon, Madeline. 2025. "Evaluating a Whatsapp based Self Development Program to Enhance English Language Skills and Teacher Agency in Pakistan." AEA RCT Registry. AEARCTR-0016340. https://doi.org/10.1257/rct.16340-1.1
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This document describes planned analysis to evaluate the impact of Beaj Education’s Self-Development course on English proficiency, psychological well-being, and career aspirations among teachers in low-cost schools across Pakistan.
Intervention
Beaj Education’s Self-Development course (“the course”) was designed in early 2025 with the goal of improving English proficiency and psychological well-being for teachers in low-cost schools across Pakistan. The content for the course was created by experts in the field of English language instruction (with 20+ years of experience), and Leadership/Psychological Coaching (with 15 years of experience).
The course included 12 weeks of English language proficiency and psychological wellness content, split into three four-week “levels”, and delivered primarily through self-paced videos. In general, each daily lesson had 6 “activities” of different types (listen and speak, watch and speak, multiple choice questions, etc.). As a secondary way of engaging with the course, all participants were assigned to a moderator-led group with up to 50 participants. All interactions within these groups took place over a group Whatsapp chat. Moderators would use these chat threads to nudge participants to engage, and some participants would share what they had completed, but these chat groups are not considered a primary form of engagement with the course.
One version of the course also integrates a bilingual (English and Urdu) and LLM-powered voice chatbot on WhatsApp, designed to serve as both an English language practice partner, personalized tutor, and personal self-development coach or mentor.
Experimental design
Nearly 3,000 teachers participated in the present study, each randomly assigned to a control group or one of two treatment groups. A baseline survey was conducted in late January 2025, the course ran between January and May 2025, and an endline survey was fielded in May 2025. The baseline and endline surveys were both conducted by phone.
In January 2025, Beaj contacted over 4,710 teachers to assess interest and availability in participating in the course. Of the 3,490 individuals who completed the baseline survey, 2,931 participants gave verbal consent over the phone to join the course. These 2,931 teachers who completed the baseline survey and signaled their intention to join and complete the course are considered as the sample for the present study. After providing verbal consent, these participants were randomly assigned to one of three groups (on a rolling basis, in six batches):
Treatment group 1 (T1): Self-Development course without AI voice chatbot
Treatment group 2 (T2): Self-Development course with AI voice chatbot
Control group (C): No access to the course
This study seeks primarily to quantify the impact of Beaj’s Self-Development course on English proficiency and psychological well-being among teachers in low-cost private schools in Pakistan, and to test the additional impact of an AI-powered voice chatbot. We hypothesize that teachers who participate in the course will demonstrate improved English proficiency and psychological well-being. We expect effects to be stronger within treatment group 2 (where an AI-powered voice chatbot complements the basic content) relative to treatment group 1 (basic content). Findings from this study will inform future programming and guide Beaj’s future scale up and engagement strategy with key stakeholders.
Note: To date, incoming data has only been analyzed for data quality purposes (tracking outreach and survey completion, checking for expected distribution of variables, assessing balance across treatment groups, etc.). No members of the research team have done any analysis related to constructing the outcomes of interest described below or estimating treatment effects. No such analysis will be conducted until after this document is filed on the AEA registry.
Design details
Cited by
No citation data available for this registration yet.