Pocket Expert or Placebo? Evaluating LLM Assistance in Second-Hand Marketplace Decisions

How to cite

DuBois, Declan; Guzman, Jakina. 2026. "Pocket Expert or Placebo? Evaluating LLM Assistance in Second-Hand Marketplace Decisions." AEA RCT Registry. AEARCTR-0019342. https://doi.org/10.1257/rct.19342-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Declan DuBois* , Jakina Guzman

* Primary investigator

Registry ID
AEARCTR-0019342
DOI
https://doi.org/10.1257/rct.19342-1.0
First registered
Last updated
Status
In development
Trial dates
2026-09-01 – 2026-09-07
Countries
Private
JEL codes
D00, D12, D18, D81, D83, C91, L81
Secondary IDs
Amherst College IRB #26-022
Registry record
http://www.socialscienceregistry.org/trials/19342

Abstract

This study evaluates whether access to a natural-language large language model assistant improves consumer decision-making and scam avoidance in simulated online second-hand marketplaces. Adult participants recruited through Prolific will complete nine marketplace choice tasks. In each task, participants will evaluate five listings for a standardized used consumer electronic product and must select the listing they perceive to be the best available purchase.

Participants will be randomly assigned at the individual level to one of two conditions: an AI-assistance treatment condition or a no-assistance control condition. Participants in the treatment condition will be permitted to consult an LLM-based assistant of their choosing while evaluating the listings. Participants in the control condition will evaluate listings generated through the same prespecified procedures without access to an assistant or other external resources.

Listings will differ in prespecified characteristics associated with transaction risk, including seller reputation, urgency language, implausible discounts, off-platform communication or transaction requests, poor product photographs, and unsecured payment methods. The study is designed to determine whether AI assistance causes consumers to respond to these signals more similarly to a benchmark sample of hired scam experts.

The observed primary outcome is whether each displayed listing is selected. A conditional logistic-regression model will estimate the effects of the prespecified listing flags for consumers assigned to AI assistance, consumers assigned to the no-AI control condition, and the expert benchmark sample while accounting for the one-of-five choice structure of each task. Average marginal effects on listing-selection probability will be the preferred common-scale representation. For each flag, the analysis will test whether the average marginal effect differs between AI-assisted consumers, control consumers, and experts. These comparisons will indicate whether AI assistance shifts consumer responses toward the pattern observed among experts. Conditional-logit coefficients will be retained as a secondary representation.

Secondary outcomes include expert-consistent choice, decision confidence, decision time, assistant use, and prespecified treatment heterogeneity by marketplace experience, product familiarity, prior AI use, and other participant characteristics.

The principal consumer sample will contain approximately 100 participants, randomized approximately equally between the two conditions. The study will be administered through Qualtrics, with consumers recruited through Prolific and experts recruited through Upwork. The expert sample is a nonrandomized benchmark group rather than a third treatment condition.

Intervention

Consumer participants will be randomly assigned to one of two conditions. Participants assigned to the AI-assistance condition will be permitted to use a large language model (LLM) assistant of their choosing while completing the marketplace choice tasks and may interact with that assistant as they wish. Participants assigned to the control condition will complete the same marketplace choice tasks without the use of AI assistance or other external information sources. All other aspects of the choice environment and listing-generation procedure will be held constant across the two consumer conditions.

Experimental design

The principal experiment is an individually randomized, two-arm online experiment. Each consumer will complete nine marketplace choice tasks via Qualtrics on a fictional online retailer, ClaCo.com. Three tasks will involve a Nintendo 3DS, three will involve a Lenovo ThinkPad laptop, and three will involve a Dell P2417H monitor. Each task will display five listings, and participants must select exactly one listing.

Listings will be drawn from product-specific pools of 128 prespecified listing variants. Each listing represents a particular combination of prespecified listing flags, such that listings may differ from one another in the number and types of flags they contain. Within each task, participants will compare the five displayed listings as a choice set and select the listing they judge to be the best option overall. The analysis will use the exact flag names and definitions contained in the final listing-level coding database. Current flag categories include seller reputation, urgency language, implausible discount cues, off-platform communication or transaction requests, poor product photographs, and unsecured payment methods.

For each task, Qualtrics will display a randomized subset of five listings from the relevant 128-listing product pool. The selected listing will be identified through its fixed answer-choice recode value. The randomized viewing-order data will also be exported so that the identities and display positions of all five listings shown to each participant can be reconstructed.

The same listing-generation procedure will be used in both consumer treatment arms. Experts will evaluate listings drawn from the same underlying product-specific pools but will complete more choice tasks than consumer participants in order to improve the precision of the expert benchmark estimates. Expert responses will be used to estimate benchmark relationships between listing flags and purchase decisions.

Design details

Randomization method
Consumer participants will be randomly assigned with equal probability to the AI-assistance or control condition using the Qualtrics Survey Flow Randomizer.
Randomization unit
The unit of randomization is the individual consumer participant.
Clusters
100 individual consumer participants. 10 experts
Observations
100 individual consumer participants. Each participant will complete nine marketplace choice tasks, yielding 900 consumer choice-task observations and 4,500 displayed listing-level observations. Each of the 10 experts will complete 27 marketplace tasks, yielding 270 consumer choice-task observations and 1,350 displayed listing-level observations.
Arms
50 participants assigned to the AI-assistance treatment condition and 50 participants assigned to the no-assistance control condition. A separate non-randomized benchmark sample of 10 experts will also be recruited and is not part of the randomized consumer treatment assignment.
Primary outcomes
Listing selection. For each listing displayed in a marketplace choice task, an indicator equal to 1 if the participant selects that listing and 0 otherwise.

Cited by

No citation data available for this registration yet.

Keywords