Human Oversight of AI Redistributive Decisions - Study 2
How to cite
Paoli, Damiano. 2026. "Human Oversight of AI Redistributive Decisions - Study 2." AEA RCT Registry. AEARCTR-0019498. https://doi.org/10.1257/rct.19498-2.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Artificial intelligence is increasingly integrated into high-stakes decisions—investment, hiring, healthcare, military targeting—making human oversight both practically important and increasingly mandated by regulation. Whether people supervise an autonomous artificial agent differently from another human is therefore a crucial question: excessive scrutiny undermines the efficiency gains that motivated AI integration, while overreliance defeats the purpose of the human-in-the-loop.
This project investigates whether individuals exhibit AI aversion when overseeing decisions with real consequences for others, and disentangles two mechanisms: the black-box effect, arising from uncertainty about the AI's decision-making process, and intrinsic AI aversion, a reluctance to rely on algorithmic judgment. In a completed first study (pre-registration at https://www.socialscienceregistry.org/trials/16680), impartial reviewers oversee redistributive decisions made by a human or an AI spectator under incomplete information. In that context, I find no evidence of AI aversion: reviewers are no more likely to overrule an AI's decision than a human's, and beliefs about the AI's decision-making process do not affect intervention.
That design tests both mechanisms at once. The second study, pre-registered here, focuses on intrinsic AI aversion: reviewers now observe the source of inequality before deciding, so an allocation that conflicts with their fairness ideal is unambiguously unfair, and any difference between overseeing a human and overseeing an AI can be attributed to a reluctance—or a preference—to defer to algorithmic judgment per se.
Intervention
Participants evaluate redistributive allocations that determine other people’s payoffs. Each allocation is labeled as being made by either a human or an AI system. Participants can either accept the allocation or pay a small fee to obtain additional information and revise it. The intervention is the different nature of the decision-maker (human vs AI) whose choice is evaluated.
Experimental design
The experiment has three parts. First, workers earn money in a real-effort task. Second, spectators decide how to redistribute earnings within a randomly drawn worker pair. Third, reviewers observe a spectator's decision and decide whether to pay a fee to change the allocation. Workers are then paid based on the final allocation. The study focuses on reviewers' decisions; workers and spectators create a consequential economic environment. Unlike the first study, reviewers act under complete information: they observe the source of inequality before deciding.
Workers. 600 workers are recruited on Prolific. After a real-effort task they are randomly paired; in each pair one worker receives an additional 4 USD, determined either by Merit (higher performance) or by Luck (random draw), with equal probability across pairs. They are informed that a third party (the spectator) will see the initial earnings and the criterion and may redistribute earnings within the pair, and that a reviewer may subsequently change the spectator's allocation.
Spectators. There are two spectator types: human (300 participants recruited on Prolific, paid a fixed 5 USD) and AI (300 independent artificial agents). Both receive identical instructions and information. Spectators face a binary choice: leave the unequal allocation (4,0) or split the earnings evenly (2,2). Each spectator decides for two pairs, one in the Merit condition and one in the Luck condition, in randomized order.
Reviewers. The same 300 human participants act as reviewers immediately after their spectator decisions, within the same session. They observe the earnings allocation, the source of inequality (Merit or Luck), and the spectator's type (human or AI), and choose between accepting the allocation and paying a fixed fee of 0.30 USD, deducted from their own earnings, to change it to the alternative allocation. Choices are elicited with the strategy method over the four criterion–allocation combinations, for both spectator types, so each reviewer decides eight times. One of each participant's ten decisions (two as spectator, eight as reviewer) is randomly selected for payment and is consequential for a pair of workers. Participants remain anonymous and are never matched to their own spectator decisions.
Design details
Cited by
No citation data available for this registration yet.