Can Artificial Intelligence Improve Writing? Experimental Evidence on AWE-Based ed-techs

How to cite

Riva, Flávio; Ferman, Bruno; Lima, Lycia. 2018. "Can Artificial Intelligence Improve Writing? Experimental Evidence on AWE-Based ed-techs." AEA RCT Registry. AEARCTR-0003729. https://doi.org/10.1257/rct.3729-2.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Flávio Riva* , Bruno Ferman , Lycia Lima

* Primary investigator

Registry ID
AEARCTR-0003729
DOI
https://doi.org/10.1257/rct.3729-2.1
First registered
Last updated
Status
Completed
Trial dates
2019-03-01 – 2019-11-30
Countries
Brazil (Espírito Santo)
JEL codes
I21, I25, I28, J22, J45
Registry record
http://www.socialscienceregistry.org/trials/3729

Abstract

Automated writing evaluation (AWE) systems use artificial intelligence to score and comment essays. We designed an experiment with two treatment arms in Brazil to study the effects of AWE-based pedagogy programs on writing skills. In both treatments, teachers were stimulated to use an AWE system that provides students with instantaneous performance signals on their essays. In both cases, students receive a final grade and formative feedback. However, in one of them this grade is set by human graders, who supervise the feedback and deliver a delayed but arguably richer assessment. The mechanisms we describe range from changes in the amount of training to the reallocation of teachers' time to different tasks. The results help address the question of whether and how these technologies can be used to improve writing skills in a post-primary education developing country context. More generally, we provide information on the potentials and limitations of artificial intelligence.

Intervention

Both treatment arms teachers were stimulated to use an Automated writing evaluation (AWE) system that provides students with instantaneous performance signals on their essays. The first treatment arm (standard treatment) uses an algorithm to provide students with instantaneous feedback on syntactic text features --- such as spelling mistakes --- and with a noisy signal of student achievement, a performance bar with 5 levels. About three days after submitting their essays on the program's platform, students receive a final grading elaborated by human graders hired by the implementer, who correct the essays trying to mimic the real-world exam. This grading includes the final essay grade, comments on the skills valued in the exam and a general comment on essay quality. In the second treatment arm (alternative treatment), the whole experience with the writing task is completed at once, and is based only on interactions with the artificial intelligence: after submitting the essays, students receive the instantaneous feedback on text features and the noisy signal of achievement (as in the first treatment arm), but are also presented to the AWE-predicted final grade and to comments selected in the implementers' database among a list of specific comments suited for a skill score. In both treatment arms, the essays and the aggregate and individual grading information generated throughout the year --- by the artificial intelligence supervised by human graders in the standard treatment, and only by the artificial intelligence in the alternative treatment --- are presented to teachers on a personal dashboard.

Experimental design

The evaluation design will be based on the random allocation of a sample of 178 public schools in Espírito Santo state into one of the three conditions for the 2019 academic year: (i) control (68 schools); (ii) standard treatment (55 schools) and (iii) alternative treatment (55 schools). More details on the interventions can be found above.

Design details

Randomization method
Randomization done in office by a computer
Randomization unit
School
Clusters
178 schools
Observations
approximately 20,000 students
Arms
Control schools: 68 Standard treatment: 55 Alternative treatment: 55
Primary outcomes
Achievement in the argumentative essay in the National Secondary Education Exam (“Exame Nacional do Ensino Medio”, ENEM).
Secondary outcomes
Mechanisms: number of essays written to train for the real ENEM essay; amount, speed, and quality of feedback; students' aspirations to enter in a post-secondary education institution; teachers' time allocation. Secondary outcomes: general writing skills; achievement in language (non-writing) subjects; achievement in non-language subjects Outcomes for follow-up papers: enrollment and achievement in post-secondary education; labor market outcomes.

Cited by

No citation data available for this registration yet.

Keywords