AI, implementation and academic outcomes in poor resource settings: experimental evidence from Ghana

How to cite

Asiedu, Prof. Edward; Henkel, Owen; Horne-robinson, Hannah. 2024. "AI, implementation and academic outcomes in poor resource settings: experimental evidence from Ghana." AEA RCT Registry. AEARCTR-0014476. https://doi.org/10.1257/rct.14476-1.3

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Prof. Edward Asiedu* , Owen Henkel , Hannah Horne-robinson

* Primary investigator

Registry ID
AEARCTR-0014476
DOI
https://doi.org/10.1257/rct.14476-1.3
First registered
Last updated
Status
In development
Trial dates
2025-01-01 – 2025-12-30
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/14476

Abstract

Low-and-middle income countries (LMIC) are experiencing a learning crisis, with less than 15% of sub-Saharan African students reaching minimum mathematics proficiency by the end of middle school (UNESCO, 2017). Students are often taught content far above their ability level and lack opportunities for practice and timely feedback. While high-quality one-on-one instruction has long been advocated to address these issues (Bloom 1984; Chi et al., 2001), the low supply and high cost of quality tutors in West Africa, pose significant barriers to access (Bray 2021). Consequently, research is needed to identify affordable and scalable interventions tailored to this context. Leveraging technology-supported learning, such as Intelligent Tutoring Systems (ITS) (Vanlehn, 2011) has shown promise in addressing diverse learning needs in LMICs. However, students in LMICs often don’t have access to home computers and the internet, though mobile phone usage is high (International Telecommunication Union 2022). The University of Ghana and Rising Academies, an educational network based in West Africa, seek to improve learning in primary and junior high schools in Ghana. Rising Academies has developed an AI-driven solution, 'Rori' to help children improve their math performance. Rori is a chat-based math tutor on WhatsApp, which can be accessed on affordable mobile phones. Rori has the potential to generate benefits for students living in poverty who cannot afford extra in-person tutorials. There are many possible avenues to using Rori as it does not require extensive bandwidth or large amounts of costly data, unlike platforms that provide video lessons. Building on encouraging results with students attending Rising Academy schools (Henkel et al., 2024), our proposed pilot will encompass a broader spectrum of schools (low-cost private schools and government-run public schools) to deepen the understanding of implementation challenges and intervention effectiveness. If Rori effectively improves math proficiency in all school types, it could support policy change and the widespread adoption of Rori, contributing to poverty reduction through enhanced educational outcomes. This pilot will also support the planning of a future RCT by helping us understand how to implement Rori in diverse contexts, ultimately contributing to developing scalable and sustainable solutions for improving educational outcomes in LMICs.

Intervention

The intervention involves providing mobile phones with Rori embedded to treatment classes. Students in grades 6-9 will participate in this pilot and all participants will receive their regular mathematics instruction. The mobile phones will be stored at the school and students will be supervised in using Rori for at least 1 hour a week.

The central research question guiding our pilot is: "How does the effectiveness of Rori vary based on implementation fidelity?". Building upon two years of research within Rising Academy schools, which has produced some promising results, we now hope to understand what is necessary to scale Rori to other schools. Therefore, this pilot aims to investigate the impact of Rori in government schools, as they are the most common type of schools in Ghana, and other low-cost private schools to compare to Rising schools. The previous evaluation of Rori was conducted with rigorous oversight in controlled environments, where implementation leads ensured adherence through frequent monitoring visits. Before investing in a large-scale RCT of Rori, it is crucial to determine if other schools can adopt Rori without that level of oversight and, if they do so, does it impacts students’ learning. Our pilot will also explore sub-questions, including the influence of school type on Rori usage, assessing the operational effectiveness of device management approaches, and teachers' attitudes and receptiveness towards the intervention. By addressing these questions, we seek to advance understanding of how to effectively implement Rori in a variety of schools, thus informing future efforts to scale up the intervention and assess its impact in a full RCT involving even more schools. We hypothesize that Rori will be effective in improving mathematics learning gains among students in the treatment group. However, we believe implementation fidelity will vary amongst the schools and thus hypothesize the size of learning gains will vary accordingly. Using a mixed-methods approach will help us determine under what conditions students have good learning gains from using Rori, again informing us how to best implement and evaluate Rori in the future.

Experimental design

The study includes students from grades 6 to 9 across Rising Academy schools, low-cost private schools, and government schools in the same neighborhood to aid comparison. We recruited 4 low- cost private schools and 5 public schools in addition to the 8 Rising Academy schools, making a total of 17 schools. Given current enrollment estimates, we expect the sample to be around 2531 students. The Rising Academy schools have been previously randomized into control and treatment. For the other schools, to avoid putting an entire school in control, we will randomize classes into treatment and control at the grade level. For example, the control for grade 6 in one school will be grade 6 in another school. Students in the control classes will not be provided with the phones with Rori. Thus, the randomization is at the classroom level within a grade.

Design details

Randomization method
For government and other private schools, randomization is done by balloting within grades across schools. For Rising Academy schools, randomization was previously for the pilot at the school level, matching treatment schools geographically with a control school. This randomization will be maintained for operational reasons.
Randomization unit
Classes
Clusters
17 schools, Four (4) grades (Class 6 to JHS 3), one class per grade 68 classrooms
Observations
We plan to assess approximately 2531 students between Grades 6 and 9 across 17 schools. We will observe and document any form of attrition across conditions, classes and schools carefully.
Arms
Treatment ("Rori"): 1260 Control ("No-Rori"): 1271
Primary outcomes
Increase in learning gains

Cited by

No citation data available for this registration yet.

Keywords