How Context Shapes Teachers’ Use of AI Grading

How to cite

Megalokonomou, Rigissa; Goulas, Sofoklis; Sotirakopoulos, Panagiotis. 2025. "How Context Shapes Teachers’ Use of AI Grading." AEA RCT Registry. AEARCTR-0016662. https://doi.org/10.1257/rct.16662-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Rigissa Megalokonomou* , Sofoklis Goulas , Panagiotis Sotirakopoulos

* Primary investigator

Registry ID
AEARCTR-0016662
DOI
https://doi.org/10.1257/rct.16662-2.0
First registered
Last updated
Status
In development
Trial dates
2025-09-21 – 2026-01-31
Countries
Greece ()
JEL codes
I21, I28, D83, C91, C93, O33
Registry record
http://www.socialscienceregistry.org/trials/16662

Abstract

Schools are beginning to deploy artificial-intelligence (AI) tools that suggest grades or feedback, yet little is known about how context and explanations shape teachers’ use of these recommendations. We study this with an anonymous, preregistered online survey experiment of K–12 teachers. Participants grade brief, realistic vignettes and are randomly assigned to (i) no AI suggestion (control), (ii) an AI grade recommendation, or (iii) the same recommendation accompanied by task/rubric context and a concise rationale (explainability). For a subset of vignettes we manipulate the accuracy of the AI suggestion, enabling tests of calibration (accepting correct and rejecting incorrect advice). Primary outcomes are AI uptake/override, accuracy-conditioned uptake, perceived trust, fairness/transparency, workload, and the between-teacher variance of assigned grades. We further explore heterogeneity by experience, subject area, and prior AI use. The design isolates whether contextual and explanatory cues help teachers discriminate between good and bad AI advice and whether such cues improve perceptions without increasing cognitive burden. Findings will inform the design and training of human-in-the-loop assessment systems that support—rather than substitute for—professional judgment.

Intervention

K–12 teachers complete a short, anonymous online survey (~9 minutes) with brief grading vignettes. For each vignette, teachers are randomly assigned to one of three conditions: (1) Control – no AI input; (2) AI recommendation – an on-screen suggested grade; (3) AI + context/explanation – the same suggestion plus brief task/rubric context and a one-sentence rationale. Teachers choose a grade and answer a few items on usefulness, fairness/transparency, and workload. No identifying information is collected, and no real student records are involved. Participation is voluntary.

Experimental design

Experimental Design (Public)
Anonymous online survey–experiment with K–12 teachers (~9 minutes). Each participant grades several short, realistic student-work vignettes. For each vignette, the platform randomly assigns one of three conditions: (i) Control (no AI input), (ii) AI recommendation (an on-screen suggested grade), or (iii) AI + context/explanation (the same suggestion plus brief task/rubric context and a one-sentence rationale). Order of vignettes is randomized.

Teachers choose a grade and answer a few brief questions on usefulness, fairness/transparency, and workload. Primary comparisons are across the three conditions (intent-to-treat). No identifying data are collected and no real student records are involved; participation is voluntary.

Design details

Randomization method
Randomization by Qualtrics
Randomization unit
individuals
Clusters
hopefully 1500 teachers
Observations
hopefully 1500 teachers
Arms
500 teachers in each arm
Primary outcomes
teacher grade

Cited by

No citation data available for this registration yet.

Keywords