Measuring Economic Determinants of AI Use in Online Data Markets

How to cite

Kuruvila, Tara Mary; Naidu, Suresh; Song, Lena. 2026. "Measuring Economic Determinants of AI Use in Online Data Markets." AEA RCT Registry. AEARCTR-0018491. https://doi.org/10.1257/rct.18491-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Tara Mary Kuruvila* , Suresh Naidu , Lena Song

* Primary investigator

Registry ID
AEARCTR-0018491
DOI
https://doi.org/10.1257/rct.18491-1.1
First registered
Last updated
Status
In development
Trial dates
2026-04-29 – 2026-12-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/18491

Abstract

This project assesses the extent to which bots and LLMs are used in the online market for survey responses, and how survey payments and performance-based bonuses affect adverse selection (low-quality or automated respondents entering the survey sample) and moral hazard (human workers using LLMs instead of providing authentic responses). We further test whether technical prevention tools and moral persuasion & penalties reduce LLM-generated responses.

Experimental design

We recruit approximately 3,000 participants aged 18–65 in the United States from Prolific, with comparison samples from other online survey platforms. Each participant completes a survey consisting of a screener and consent form, demographic questions, a battery of incentivized tasks, and a post-task section on self-reported AI use and attitudes. The task battery spans probability reasoning, beliefs and attitudes questions, open-ended summarization, and data labeling, reflecting the range of activities common in behavioral research and machine-learning data work. We cross-randomize four dimensions: base payment (high/low), bonus payment (none/low/high), prevention tools designed to discourage AI use (present/absent), and sanctions for AI use (none/moral persuasion and monetary penalty). Base payment is randomized at the study-listing level; the remaining dimensions are randomized at the individual level. To detect AI use, we combine behavioral measures (keystrokes, mouse movement, copy-paste activity, tab-switching, time on page), text-based AI detection on open-ended responses, and self-reported AI use, aggregated into pre-specified indices. After the study is complete, we invite participants to complete a follow-up survey to measure retention. For comparison, we also run this study on other platforms with design depending on budget and feasibility.

Design details

Randomization method
In qualtrics-survey randomization. Exception is base pay which is done at the listing level.
Randomization unit
Individual; Listing-level for base-pay.
Clusters
ca. 4000 online survey participants across multiple platforms
Observations
ca. 4000 online survey participants across multiple platforms
Arms
We cross-randomize four dimensions in a 2 × 3 × 2 × 2 factorial design (24 conditions): base payment (high/low, randomized at the study-listing level) and bonus payment (none/low/high), prevention tools (present/absent), and sanctions (none/moral persuasion and monetary penalty). There are an equal number of participants in each treatment cell.
Primary outcomes
• AI use as detected by each of our methods (Lasso, PCA, pre-specified binary) • Performance
Secondary outcomes
See pre-analysis plan

Cited by

No citation data available for this registration yet.

Keywords