AI Recommendations and Incentives Under Time Constraints: Evidence from an RCT

How to cite

Chupak, Maxwell. 2026. "AI Recommendations and Incentives Under Time Constraints: Evidence from an RCT." AEA RCT Registry. AEARCTR-0018623. https://doi.org/10.1257/rct.18623-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Maxwell Chupak*

* Primary investigator

Registry ID
AEARCTR-0018623
DOI
https://doi.org/10.1257/rct.18623-1.0
First registered
Last updated
Status
Completed
Trial dates
2026-08-13 – 2026-08-20
Countries
United States of America ()
Registry record
http://www.socialscienceregistry.org/trials/18623

Abstract

This paper studies how the size of the monetary penalty for inaccurate responses affects subjects' reliance on AI-generated recommendations. Subjects complete a series of CAPTCHA-style object-counting tasks, with and without access to a pre-generated recommendation from an AI system, permitting within-subject comparison of task accuracy and AI reliance across AI-available and AI-unavailable conditions; this contrast is implemented under one of three between-subjects penalty levels for incorrect responses. To help explain over- or under-reliance relative to the AI's true, undisclosed accuracy, we separately elicit subjects' subjective belief distribution over the AI's accuracy using an incentive-compatible quadratic scoring rule instrument (Matheson and Winkler, 1976), and their willingness to pay for AI access using a Becker-DeGroot-Marschak (1964) mechanism. Subjects' subjective beliefs about the AI's accuracy are compared to their own self-assessed and task-measured accuracy to provide further evidence on why subjects may over- or under-rely on AI recommendations (Moore and Healy, 2008). Subjects are recruited via Prolific and complete the experiment on Qualtrics.

Intervention

Subjects complete three modules, each containing 30 unique judgment tasks. Subjects are randomly assigned to one of three financial incentive conditions that differ in the monetary penalty for incorrect responses. AI recommendations are available in exactly one of the first two modules and unavailable in the other; this availability is varied within-subject and counterbalanced across subjects. Access to AI recommendations in the third module is instead determined by a Becker-DeGroot-Marschak (BDM) mechanism eliciting each subject's willingness to pay. In every module, the first task is one for which the associated AI answer is fully accurate, and the second task is one for which the associated AI answer is off by one; the relative order of these two tasks is randomized independently at each module load, regardless of whether AI recommendations are available in that module. All remaining tasks are presented in a fixed, predetermined order. When an AI recommendation is available, subjects may reveal it and, with an additional click, incorporate it into their response, so that viewing and adopting a recommendation are captured as distinct decisions.

Experimental design

Participants complete three modules of incentivized recognition tasks under time constraints. AI recommendations are available in one of the first two modules, with AI availability varied within subjects and module order counterbalanced across participants. Participants are randomly assigned to one of three financial incentive conditions that differ in the penalty for incorrect responses. In the third module, access to AI recommendations is determined through a Becker-DeGroot-Marschak (BDM) mechanism eliciting willingness to pay. Participants also complete an incentivized elicitation of their beliefs regarding the AI’s accuracy. Earnings are determined by one randomly selected module.

Design details

Randomization method
Randomization of first task accuracy, module presentation order, BDM price, and assignment to penalty level is conducted by the Qualtrics survey software.
Randomization unit
Penalty level is randomized at the subject level. First task accuracy for modules with AI recommendations and module order across the experiment are randomized within subject.
Clusters
Not applicable. Randomization occurs at the individual participant level.
Observations
400 individual participants. Up to 36,000 task-level observations (400 participants × 90 tasks), conditional on completion of all tasks..
Arms
Approximately 133 participants per penalty treatment arm.
Primary outcomes
For each subject, the primary outcomes are 1. The within-subject difference between AI and non-AI modules in number of tasks answered correctly, number of completed tasks, and percent of tasks correctly completed. 2. The subject’s willingness to pay for access to AI recommendations for the final module. Behavioral measures of reliance (e.g., AI-recommendation reveal and adoption rates) are analyzed as secondary outcomes; in the absence of a structural model dictating a single theoretically-preferred reliance measure, we do not designate any one specification as primary.
Secondary outcomes
Secondary outcomes include, 1. The mean of each subject’s elicited subjective belief distribution over the AI’s accuracy. 2. The deviation between the mean of each subject's elicited belief distribution and the AI’s true accuracy. 3. The deviation between the mean of each subject's expected own accuracy and the subject's true accuracy. 4. Behavioral measures of reliance (e.g., AI-recommendation reveal and adoption rates) are analyzed as secondary outcomes; in the absence of a structural model dictating a single theoretically-preferred reliance measure, we do not designate any one specification as primary.

Cited by

No citation data available for this registration yet.

Keywords