The What, When and Why of AI Use in Online Preference Elicitation Experiments

How to cite

Morozova, Anastasiia; Brooksby, Austin. 2025. "The What, When and Why of AI Use in Online Preference Elicitation Experiments." AEA RCT Registry. AEARCTR-0016965. https://doi.org/10.1257/rct.16965-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Anastasiia Morozova* , Austin Brooksby

* Primary investigator

Registry ID
AEARCTR-0016965
DOI
https://doi.org/10.1257/rct.16965-1.0
First registered
Last updated
Status
In development
Trial dates
2025-10-14 – 2025-12-16
Countries
Private
JEL codes
C90, D91
Registry record
http://www.socialscienceregistry.org/trials/16965

Abstract

We explore the effects of AI in online preference elicitation experiments (risk, time, and social preferences) by providing subjects with an LLM within the experiment. AI provision itself does not substantially change the outcomes in all of our preference types, but the AI use has a noticeable impact on tasks that seem less intuitive to participants and tasks where AI can anchor the advice to a particular number. The measures of usefulness of AI provided by the participants are highly correlated with the measures of change in risk preferences in response to AI advice. We argue that specific elicitation methods cause more confusion and are more prone to being delegated to AI by the experimental subjects. We also point out particular patterns of AI use that can allow researchers to AI-proof their experiments.

Intervention

We conduct canonical versions of risk, time, and social preferences elicitation tasks. We deploy these tasks in conjunction with a treatment, providing treated participants with the option to chat with an LLM within the experiment.

Experimental design

We run three experimental sessions on Prolific: separately for risk, time, and social preferences elicitation. Participants are randomized in control (no LLM) or treatment (LLM offered) group.

Design details

Randomization method
Done by computer
Randomization unit
Subject
Clusters
NA
Observations
128 subjects for risk elicitation, 100 subjects for social preferences
Arms
Risk: 64 subjects in treatment, 64 subjects in control. Social preferences: 50 people in treatment, 50 people in control
Primary outcomes
Risk elicitation: Risk preference parameters (assuming CRRA) and absolute deviation from expected value. Time preference elicitation: time preference parameters under hyperbolic discounting. Social preferences: in trust game -- amount sent, reciprocity amounts, in dictator game -- amount sent, in ultimatum game -- proposer's offer and minimum acceptable offer. The comparison between treatment and control group for these key variables constitutes the intent-to-treat effect of AI availability in online preference elicitation experiments. The proportion of subjects utilizing the LLM chat across experiments, in conjunction with measures of task-specific subjective AI value, as a measure of elicitation method-induced subject confusion. The take-up of AI option allows us to construct the treatment-on--the-treated measure of AI impact on preference elicitation in online experiments.
Secondary outcomes
For all decisions, we elicit the level of certainty in the choice, the time to reach the decision, the perceived helpfulness of AI, and the belief about how intuitive the task was. We also elicit the subject's willingness to pay for AI to make all decisions in the task and their willingness to pay for the AI to revise the decisions made by the subject, both before and after the task's completion. For the treatment group, we record the use of the LLM (take-up rate and chat history).

Cited by

No citation data available for this registration yet.

Keywords