The Effect of Giving Instructors Agency to Choose Among Different Types of Automated, Natural Language Processing Based Feedback

How to cite

Demszky, Dora; Hill, Heather; Taylor, Eric; Piech, Chris. 2023. "The Effect of Giving Instructors Agency to Choose Among Different Types of Automated, Natural Language Processing Based Feedback." AEA RCT Registry. AEARCTR-0012746. https://doi.org/10.1257/rct.12746-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Dora Demszky* , Heather Hill , Eric Taylor , Chris Piech

* Primary investigator

Registry ID
AEARCTR-0012746
DOI
https://doi.org/10.1257/rct.12746-1.0
First registered
Last updated
Status
Completed
Trial dates
2023-04-01 – 2023-06-22
Countries
United States of America ()
Registry record
http://www.socialscienceregistry.org/trials/12746

Abstract

This project builds on a previous study conducted in 2021 on Code in Place, a five-week-long online programming course where we found that automated feedback to instructors can improve their instruction and student satisfaction. The current study was conducted in the spring of 2023 on Code in Place, and its goal is to understand whether providing instructors agency in choosing the type of automated feedback they would like to receive influences their engagement with and impact of the feedback. Learner agency is thought to enhance engagement and improve outcomes, but few empirical studies have examined its effect in instructor learning settings. To answer this question, the study leverages both manual annotation and computational natural language processing techniques.

Intervention

Please refer to Analysis Plan document.

Experimental design

Sample size: 588 instructors
The study was conducted in a free, online 6 week long introductory programming course called Code in Place. Anyone could apply to serve as a volunteer section leader on the course, and then they were selected by the course organizers. Our participant sample consists of all adult (18+) instructors in Code in Place.

Before the course began: We randomized instructors once they were accepted to teach in the course, and before the course began. Half of the instructors got a choice for what type of automated feedback they wanted to receive (see figure below). Instructors were asked to make this choice on the Code in Place website, and this action item was listed on their pre-course checklist. The choice involved feedback on 3 types of talk moves (Getting ideas on the table, Building on student ideas, Orienting students to one another), which they could select for pairs of weeks (1-2, 3-4, weeks 5-6*). Instructors also had the option to enable experimental, GPT-4 based feedback for the last 2 weeks and had the option to compare their metrics with other section leaders for all weeks. We displayed a short definition and an example below each talk move to help inform their choices. We sent email nudges to instructors before the course began to make a choice.

*We had thought that the course would only be 5 weeks long, hence the choice interface only had Week 5 listed for the third box; as we realized the course would be 6 weeks long, we applied their choices for week 5 to week 6 as well.

The control group did not get to choose. Instead the control group was randomly assigned to feedback under the constraint that the distribution of feedback patterns in the control group was the same as the distribution in the treatment group. For example, 36% of the treatment group chose the pattern: 2 weeks on getting ideas on the table, 2 weeks on building on student ideas, 2 weeks of experimental feedback. Thus 36% of the control group were assigned to that same pattern. This assignment of the control group means that the only difference between treatment and control, in expectation, is whether the instructor chose their pattern of feedback or were assigned their pattern of feedback.

About 80% of treatment group instructors made a choice of what feedback to receive. The 20% that did not choose got were assigned feedback with a similar method as the control group.

All instructors had access to training modules that explained each talk move and showed animations to illustrate the talk move. Both control and treatment group instructors were encouraged to complete these modules prior to their first section. Completion rates were about 40% for each of the training modules.

Choice of feedback: 36% of instructors choose the following sequence of feedback: Getting ideas on the table, building on student ideas, then the experimental (GPT-4) feedback with the choice to compare their metrics to other section leaders; this sequence of feedback was by far the most popular choice. The rest of the instructors choose a combination of other feedback choices.

Once the course began: After each of their six sessions, instructors received automated feedback based on their chosen or assigned feedback type. The feedback was released to everyone on Saturday (sections taught Wed-Friday) via email. The email varied: the treatment group was reminded that they got a choice and here’s the feedback on that topic. The control group did not get this reminder that they had made a choice. Below is an example email that the treatment group received.

All instructors could view their automated feedback on the Code in Place webpage. The talk move feedback included several components:
Introduction to the feedback
Summary statistics for the given talk move
Definition of each talk move
Their talk moves in action (list of talk moves from their transcript)
Link to the relevant training module
Reflection opportunities

Design details

Randomization method
Hashing function (similar to coin flip)
Randomization unit
Instructor
Clusters
N/A
Observations
588
Arms
50% split between observations
Primary outcomes
RQ1: Engagement with feedback RQ2: Perception of feedback RQ3: Instructional practices RQ4: Student outcomes Please refer to Analysis plan for further details.

Cited by

No citation data available for this registration yet.

Keywords