THE EFFECTS OF PERSONALIZED FEEDBACK: Replication in a non-pandemic context.

How to cite

Aucejo, Esteban; Wong, Kelvin. 2023. "THE EFFECTS OF PERSONALIZED FEEDBACK: Replication in a non-pandemic context.." AEA RCT Registry. AEARCTR-0011939. https://doi.org/10.1257/rct.11939-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Esteban Aucejo* , Kelvin Wong

* Primary investigator

Registry ID
AEARCTR-0011939
DOI
https://doi.org/10.1257/rct.11939-1.0
First registered
Last updated
Status
In development
Trial dates
2023-08-23 – 2024-08-01
Countries
Private
JEL codes
I2
Registry record
http://www.socialscienceregistry.org/trials/11939

Abstract

This study examines the effects of scalable, personalized email feedback on students at a large public institution in the US. We aim to discern which feedback types most significantly influence student learning and retention, specifically identifying the content that most likely alters student behavior. Our research closely mirrors the experiment presented in the working paper “The Effect of Feedback on Student Performance” by Esteban Aucejo and Kelvin Wong.

Our analysis seeks to validate these findings in an environment where the pandemic does not influence classes (Fall 2023). The pandemic led to diminished interactions between students and professors due to the reliance on platforms like Zoom for instruction. In such a scenario, personalized feedback emails might have held greater significance for students, potentially resulting in more pronounced effects.

We aim to evaluate the varied effects of feedback messages on three distinct student groups:

a) First-generation students (those whose parents did not attend college) versus non-first-generation students,
b) Over-optimistic students (defined by the disparity between their expected class rank and their rank after the first exam) versus their non-overoptimistic counterparts,
c) Mode of instruction students received: traditional in-person, hybrid, or online classes.

To gauge the influence of feedback, we will evaluate both educational outcomes and qualitative student responses. We will explore students' perceptions of personalized feedback messages and determine how their beliefs shift post-feedback.

Intervention

Students enrolled in introductory economics courses will receive personalized feedback emails based on their performance in the initial class exam. The distribution of these emails will be randomized yet stratified by factors such as first-generation status, gender, and letter grade achieved in the first exam. Additionally, we will conduct two surveys to collect background data and gauge any shifts in students' beliefs regarding their capabilities.

Experimental design

Students from a largepublic higher education institution, specifically those enrolled in economics courses, will receive varied feedback based on their exam results. The objective is to ascertain the impact of personalized feedback on student outcomes.

Design details

Randomization method
The randomization will be done in an office by a computer using the Stata command "stratarand".
Randomization unit
Randomization will be done at the student level, conditional on attending the same class (we have approximately 20 to 23 classes in our experiment). Students will receive a feedback email at random conditional on their performance in the class and providing consent in the initial survey. Treatment assignment will be done by stratifying the sample.
Clusters
No clusters, we will randomize students into the treatment and control groups within the classroom. We plan to perform the randomization on approximately 20 to 23 classrooms. The final number of classrooms still depends on receiving consent from a couple of professors in charge of a few classrooms.
Observations
Our sample involves approximately 2000 students divided across different classrooms. The final number of students will depend on the share of them providing consent to participate in the program. Randomization will be performed on the sample of students that provided consent conditional on classroom.
Arms
Approximately half students will be randomly allocated to the treatment groups and the other half to the control group.
Primary outcomes
We intend to evaluate the following primary outcomes: 1. Performance in the second exam, 2. Final course grade, 3. Performance in the second exam and final grade in courses where no exams are dropped (given that some courses permit students to exclude an exam score), 4. Percentage of problem sets successfully completed (i.e., passed), 5. Whether the student opts to change their major or field of study, and 6. The student's perception of the usefulness of specific types of feedback.

Cited by

No citation data available for this registration yet.

Keywords