Synthetic Experiments for Decision-Making Under Uncertainty: Validating AI Agent-Based Research Methods Against Human Experimental Data

How to cite

Pandey, Abhinav; Camuffo, Arnaldo; Gambardella, Alfonso; Mariani, Myriam. 2025. "Synthetic Experiments for Decision-Making Under Uncertainty: Validating AI Agent-Based Research Methods Against Human Experimental Data." AEA RCT Registry. AEARCTR-0017171. https://doi.org/10.1257/rct.17171-4.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Abhinav Pandey* , Arnaldo Camuffo , Alfonso Gambardella , Myriam Mariani

* Primary investigator

Registry ID
AEARCTR-0017171
DOI
https://doi.org/10.1257/rct.17171-4.0
First registered
Last updated
Status
In development
Trial dates
2025-11-04 – 2026-07-31
Countries
Private
JEL codes
O33
Registry record
http://www.socialscienceregistry.org/trials/17171

Abstract

Can synthetic experiments with AI agents serve as rigorous research tools for generating insights, testing hypotheses, and informing managerial decision-making under uncertainty? This pre-analysis plan specifies a comprehensive validation study comparing 16 persona generation methods across a 2×2×4 factorial design: two base sampling algorithms crossed with a clustering algorithm and four constraint configurations. Each of the 16 persona generation methods generates a sample of 800 college undergraduate students who serve as synthetic experimental subjects. The 16 samples undergo the identical 2×2 factorial experiment (Causal Reasoning Training × ChatGPT Access) previously conducted with real human participants, with participants randomized across the four experimental conditions (200 per cell). Our research questions ask: (1) Which persona generation methods better approximate humans as experimental subjects? (2) How do the three design dimensions (base algorithm, archetypes, constraints) affect approximation quality? (3) Can process differences in manipulation checks and compliance explain divergent outcomes? By benchmarking against gold standard human experimental data, we develop methods for building high-fidelity synthetic agents and define a scalable protocol for conducting synthetic experiments in management research. This registration ensures transparency in methodology specification and analytical decisions prior to synthetic data generation.

Intervention

The synthetic experiments replicate the experimental design of a randomized controlled trial previously conducted with human subjects (pre-registered separately). The experiment with human subjects tested whether causal reasoning trainig (Intervention 1 via causal resoning game) and ChatGPT access (Intervention 2) produce complementary effects on strategic decision-making performance.

Experimental design

The synthetic experiments replicate the experimental design of a randomized controlled trial previously conducted with human subjects (pre-registered separately). The experiment with human subjects tested whether causal reasoning training and ChatGPT access produce complementary effects on strategic decision-making performance using a 2×2 factorial design with 800 participants. Using the results from the experiment with human subjects as ground truth, these 16 synthetic experiments test the same hypotheses and attempt to replicate the same results. The overarching goal is to test the extent to which synthetic agents approximate the responses of human subjects and therefore produce similar experimental outcomes. Key Design Principle: The experimental design and execution are identical across all 17 experiments (16 synthetic + 1 human). The ONLY difference is the participant pool: human subjects versus synthetic personas generated through different methods.

Design details

Randomization method
Simple randomization assigns personas to the four experimental conditions (n=200 per cell). Stratification balances on key demographic variables. Randomization seeds are logged for reproducibility.
Randomization unit
Syntetic persona
Clusters
0 clusters
Observations
800 synthetic personas
Arms
200 synthetic personas for treatment arm Non-Causal, No ChatGPT. 200 synthetic personas for treatment arm Non-Causal, ChatGPT. 200 synthetic personas for treatment arm Causal, No ChatGPT. 200 synthetic personas for treatment arm Causal, ChatGPT.
Primary outcomes
Performance score = (Awareness + Usage) / 2
Secondary outcomes
Causal reasoning score

Cited by

No citation data available for this registration yet.

Keywords