Synthetic Experiments for Decision-Making Under Uncertainty: Validating AI Agent-Based Research Methods Against Human Experimental Data
How to cite
Pandey, Abhinav; Camuffo, Arnaldo; Gambardella, Alfonso; Mariani, Myriam. 2025. "Synthetic Experiments for Decision-Making Under Uncertainty: Validating AI Agent-Based Research Methods Against Human Experimental Data." AEA RCT Registry. AEARCTR-0017171. https://doi.org/10.1257/rct.17171-4.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Can synthetic experiments with AI agents serve as rigorous research tools for generating insights, testing hypotheses, and informing managerial decision-making under uncertainty? This pre-analysis plan specifies a comprehensive validation study comparing 16 persona generation methods across a 2×2×4 factorial design: two base sampling algorithms crossed with a clustering algorithm and four constraint configurations. Each of the 16 persona generation methods generates a sample of 800 college undergraduate students who serve as synthetic experimental subjects. The 16 samples undergo the identical 2×2 factorial experiment (Causal Reasoning Training × ChatGPT Access) previously conducted with real human participants, with participants randomized across the four experimental conditions (200 per cell). Our research questions ask: (1) Which persona generation methods better approximate humans as experimental subjects? (2) How do the three design dimensions (base algorithm, archetypes, constraints) affect approximation quality? (3) Can process differences in manipulation checks and compliance explain divergent outcomes? By benchmarking against gold standard human experimental data, we develop methods for building high-fidelity synthetic agents and define a scalable protocol for conducting synthetic experiments in management research. This registration ensures transparency in methodology specification and analytical decisions prior to synthetic data generation.
Intervention
The synthetic experiments replicate the experimental design of a randomized controlled trial previously conducted with human subjects (pre-registered separately). The experiment with human subjects tested whether causal reasoning trainig (Intervention 1 via causal resoning game) and ChatGPT access (Intervention 2) produce complementary effects on strategic decision-making performance.
Experimental design
The synthetic experiments replicate the experimental design of a randomized controlled trial previously conducted with human subjects (pre-registered separately). The experiment with human subjects tested whether causal reasoning training and ChatGPT access produce complementary effects on strategic decision-making performance using a 2×2 factorial design with 800 participants. Using the results from the experiment with human subjects as ground truth, these 16 synthetic experiments test the same hypotheses and attempt to replicate the same results. The overarching goal is to test the extent to which synthetic agents approximate the responses of human subjects and therefore produce similar experimental outcomes. Key Design Principle: The experimental design and execution are identical across all 17 experiments (16 synthetic + 1 human). The ONLY difference is the participant pool: human subjects versus synthetic personas generated through different methods.
Design details
Cited by
No citation data available for this registration yet.