Detecting AI-Generated Text in Online Survey Responses

How to cite

Buschmann, Andy. 2026. "Detecting AI-Generated Text in Online Survey Responses." AEA RCT Registry. AEARCTR-0017629. https://doi.org/10.1257/rct.17629-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Andy Buschmann*

* Primary investigator

Registry ID
AEARCTR-0017629
DOI
https://doi.org/10.1257/rct.17629-1.0
First registered
Last updated
Status
Completed
Trial dates
2026-01-09 – 2026-01-12
Countries
United States of America ()
JEL codes
C8
Registry record
http://www.socialscienceregistry.org/trials/17629

Abstract

This study examines whether a syntax-based detection method based on character encoding patterns can identify AI-assisted text generation in open-ended survey responses. Specifically, it evaluates whether mixed or ASCII-only quotation mark usage, which arise from copy-pasting text generated in AI user interfaces such as ChatGPT, occurs in incentivized online survey responses and whether such patterns align with or diverge from respondents’ self-reported AI use.

Intervention

Participants are randomly assigned to one of three survey conditions that vary how artificial intelligence (AI) tools are described prior to a creative writing task. In one condition, participants are explicitly told they may use AI tools to assist with the task. In a second condition, participants are explicitly instructed not to use AI tools. In a control condition, AI tools are not mentioned prior to the task.

Experimental design

The study uses a between-subjects/split-ballot randomized survey experiment. Participants complete a creative writing task and related survey questions. The experimental manipulation consists of varying the instructional framing regarding AI tool use prior to the writing task. Outcomes are measured using both self-reports and post hoc analysis of written responses.

Design details

Randomization method
Randomization is implemented by the survey platform (Qualtrics) using a built-in computer-generated simple random assignment procedure.
Randomization unit
Individual survey respondents.
Clusters
Not applicable (individual-level randomization).
Observations
Approximately 450 individual respondents.
Arms
Approximately 150 respondents in each of three conditions: Control, AI Allowed, and No-AI Instruction.
Primary outcomes
Presence of character encoding patterns in open-ended survey responses consistent with AI-assisted text generation.
Secondary outcomes
Self-reported AI use during the writing task; response length (word and character counts); agreement between detection flags and self-reported AI use.

Cited by

No citation data available for this registration yet.

Keywords