LLMs, Compliance, and Decision-Making in Organizations: A Field Experiment

How to cite

Chen, Yutong. 2026. "LLMs, Compliance, and Decision-Making in Organizations: A Field Experiment." AEA RCT Registry. AEARCTR-0019059. https://doi.org/10.1257/rct.19059-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Yutong Chen*

* Primary investigator

Registry ID
AEARCTR-0019059
DOI
https://doi.org/10.1257/rct.19059-1.0
First registered
Last updated
Status
In development
Trial dates
2026-06-28 – 2026-11-01
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/19059

Abstract

This project conducts a field experiment at a large-scale shrimp farm in China to examine how LLMs affect compliance in organizations. Workers receive AI-generated harvest recommendations, and in one treatment they can also interact with a LLM chatbot to explore the rationale behind the recommendations. Using daily pond-level data, we study how workers respond when AI recommendations conflict with managerial directives, focusing on whether LLMs improves compliance or instead provides a justification for deviation. We further test whether engagement with the AI system strengthens its behavioral impact and whether forecast errors weaken trust in AI. By linking behavioral, engagement, and economic outcomes, the study provides causal evidence on how LLMs may reshape coordination, authority, and control in organizations. The findings speak to the broader question of how information frictions within organizations undermine managerial control.

Experimental design

We will conduct a cluster-randomized field experiment at the pond level in a large shrimp-farm setting. The study has two treatment arms: in both arms, workers receive access to the same AI forecasting tool and numerical harvest recommendations as managers; in the second arm, workers also receive a chatbox interface that allows them to ask why and discuss the rationale behind the AI recommendation. The randomization unit is the pond, and all analyses will be clustered accordingly. Data are observed at the pond-day level over the production season, allowing us to measure whether workers comply with managerial directives when AI recommendations conflict with those directives. The intervention is implemented through a website using a fine-tuned forecasting model that integrates weather, hydrological, pond, and historical operational data.

Design details

Randomization method
Randomization will be conducted at the pond level, using a randomized assignment in the research office
Randomization unit
Randomization unit: pond. The study randomizes treatment at the pond level, and standard errors are clustered at the pond level in analysis.
Clusters
12
Observations
67
Arms
67
Primary outcomes
Primary outcome: worker compliance with the manager’s directive at the pond-day level, measured by the actual harvest decision and whether it matches or deviates from the manager’s instruction. The core endpoint is the binary daily harvest outcome for each pond.

Cited by

No citation data available for this registration yet.

Keywords