Impact of introducing AI technology on students' writing skills: evidence for Brazil

How to cite

Barbosa, Rafael; Ponti, Mayra. 2023. "Impact of introducing AI technology on students' writing skills: evidence for Brazil." AEA RCT Registry. AEARCTR-0010887. https://doi.org/10.1257/rct.10887-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Rafael Barbosa* , Mayra Ponti

* Primary investigator

Registry ID
AEARCTR-0010887
DOI
https://doi.org/10.1257/rct.10887-1.0
First registered
Last updated
Status
In development
Trial dates
2023-03-15 – 2023-06-30
Countries
Private
JEL codes
I2; I25; I29
Registry record
http://www.socialscienceregistry.org/trials/10887

Abstract

In this study, we evaluated the impact of the introduction of Artificial Intelligence technology in the correction of Portuguese essays on students' writing skills. To do so, teachers in three municipalities in the state of Alagoas, a poor state in Brazil, were randomly assigned to access an online platform for three months. This platform allows teachers to upload essay texts produced by students, giving back to the teachers the students' performance in four aspects of writing skills: formal writing register, cohesion, thematic coherence, and textual typology. The objective of the experiment is to verify whether there has been an improvement in the quality of textual production of primary school students.

Intervention

The intervention will be carried out among schools in three municipalities in the state of Alagoas: Arapiraca and Girau do Ponciano. These municipalities have 43 elementary schools and approximately 3200 students in the 6th grade. Randomization will be carried out using a stratification strategy at the school level. The following variables will be used to stratify the schools: school size, measured by the school average class size, and rural vs. urban. The treatment will consist of access to the teachers of the treated schools to an online platform that allows teachers to upload their students' writing texts and return them back to them analyzed in four aspects: formal writing register, cohesion, thematic coherence, and textual typology. On this platform, besides the feedback on the corrected essays, pedagogical materials will also be available for teachers to use in the classroom.

Before the experiment starts, a baseline will be conducted with students and teachers. This baseline will consist of an essay written by the students on a topic appropriate to their grade level. Additionally, information will be collected on the socio-demographic characteristics of the students. The teachers will answer an online questionnaire that will collect information about their work routine, their self-efficacy in teaching text production, and their ICT skills. To expand the sample of teachers, this online questionnaire will be applied to all elementary school teachers in the participating schools, not only to 6th-grade teachers.

At the end of the intervention, the endline will be applied to both students and teachers. Students will perform a written production exercise similar to the baseline. And teachers will be asked again about their work routine during the intervention, use of feedback, self-efficacy in teaching writing skills, and questions related to the use of available teaching resources.

Experimental design

The experiment will be conducted in three Brazilian municipalities: Arapiraca and Girau do Ponciano

Design details

Randomization method
Randomization will be carried out using a stratification strategy at the school level. The following variables will be used to stratify the schools: performance in IDEB, and rural vs. urban.
Randomization unit
Strata
Clusters
The sample contains 43 elementary schools with approximately 3200 students and 270 teachers.
Observations
1678
Arms
22 schools in treatment, 21 schools control
Primary outcomes
Student's writing skills
Secondary outcomes
The secondary results refer to the teachers' practices when using the online platform. They will be evaluated: 1. time available for teaching activities (teaching grammar, literature, and developing writing skills) 2. Self-efficacy in the development of students' writing skills 3. Number of writing exercises

Cited by

No citation data available for this registration yet.

Keywords