Human Oversight of AI Redistributive Decisions - Study 2

How to cite

Paoli, Damiano. 2026. "Human Oversight of AI Redistributive Decisions - Study 2." AEA RCT Registry. AEARCTR-0019498. https://doi.org/10.1257/rct.19498-2.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Damiano Paoli*

* Primary investigator

Registry ID
AEARCTR-0019498
DOI
https://doi.org/10.1257/rct.19498-2.0
First registered
Last updated
Status
In development
Trial dates
2026-08-28 – 2027-03-01
Countries
Private
JEL codes
C91, D63, O33
Registry record
http://www.socialscienceregistry.org/trials/19498

Abstract

Artificial intelligence is increasingly integrated into high-stakes decisions—investment, hiring, healthcare, military targeting—making human oversight both practically important and increasingly mandated by regulation. Whether people supervise an autonomous artificial agent differently from another human is therefore a crucial question: excessive scrutiny undermines the efficiency gains that motivated AI integration, while overreliance defeats the purpose of the human-in-the-loop.

This project investigates whether individuals exhibit AI aversion when overseeing decisions with real consequences for others, and disentangles two mechanisms: the black-box effect, arising from uncertainty about the AI's decision-making process, and intrinsic AI aversion, a reluctance to rely on algorithmic judgment. In a completed first study (pre-registration at https://www.socialscienceregistry.org/trials/16680), impartial reviewers oversee redistributive decisions made by a human or an AI spectator under incomplete information. In that context, I find no evidence of AI aversion: reviewers are no more likely to overrule an AI's decision than a human's, and beliefs about the AI's decision-making process do not affect intervention.

That design tests both mechanisms at once. The second study, pre-registered here, focuses on intrinsic AI aversion: reviewers now observe the source of inequality before deciding, so an allocation that conflicts with their fairness ideal is unambiguously unfair, and any difference between overseeing a human and overseeing an AI can be attributed to a reluctance—or a preference—to defer to algorithmic judgment per se.

Intervention

Participants evaluate redistributive allocations that determine other people’s payoffs. Each allocation is labeled as being made by either a human or an AI system. Participants can either accept the allocation or pay a small fee to obtain additional information and revise it. The intervention is the different nature of the decision-maker (human vs AI) whose choice is evaluated.

Experimental design

The experiment has three parts. First, workers earn money in a real-effort task. Second, spectators decide how to redistribute earnings within a randomly drawn worker pair. Third, reviewers observe a spectator's decision and decide whether to pay a fee to change the allocation. Workers are then paid based on the final allocation. The study focuses on reviewers' decisions; workers and spectators create a consequential economic environment. Unlike the first study, reviewers act under complete information: they observe the source of inequality before deciding.

Workers. 600 workers are recruited on Prolific. After a real-effort task they are randomly paired; in each pair one worker receives an additional 4 USD, determined either by Merit (higher performance) or by Luck (random draw), with equal probability across pairs. They are informed that a third party (the spectator) will see the initial earnings and the criterion and may redistribute earnings within the pair, and that a reviewer may subsequently change the spectator's allocation.

Spectators. There are two spectator types: human (300 participants recruited on Prolific, paid a fixed 5 USD) and AI (300 independent artificial agents). Both receive identical instructions and information. Spectators face a binary choice: leave the unequal allocation (4,0) or split the earnings evenly (2,2). Each spectator decides for two pairs, one in the Merit condition and one in the Luck condition, in randomized order.

Reviewers. The same 300 human participants act as reviewers immediately after their spectator decisions, within the same session. They observe the earnings allocation, the source of inequality (Merit or Luck), and the spectator's type (human or AI), and choose between accepting the allocation and paying a fixed fee of 0.30 USD, deducted from their own earnings, to change it to the alternative allocation. Choices are elicited with the strategy method over the four criterion–allocation combinations, for both spectator types, so each reviewer decides eight times. One of each participant's ten decisions (two as spectator, eight as reviewer) is randomly selected for payment and is consequential for a pair of workers. Participants remain anonymous and are never matched to their own spectator decisions.

Design details

Randomization method
Randomization is done through the experimental software (oTree).
Randomization unit
Randomization happens at the individual level. The two spectator-type blocks are presented in randomized order, in balanced blocks: half of the reviewers (150) first evaluate the allocations made by human spectators and then those made by AI spectators; the other half face the opposite order. Within each block, the order of the four criterion-allocation combinations is randomized.
Clusters
300 individuals, acting first as spectators and then as reviewers. The sample is representative of the U.S. population in terms of sex, age, and political affiliation, as determined by Prolific's representative sample distribution.
Observations
2,400 reviewer observations = 300 individuals × 4 criterion-allocation combinations × 2 spectator types.
Arms
150 participants (600 observations) with AI spectators first, 150 participants (600 observations) with human spectators first (between-subjects).
Primary outcomes
The main outcome is the intervention rate. Intervene(i,a): dummy variable equal to 1 if the reviewer decides to pay the fee and change the allocation to the alternative one, and 0 otherwise. The choice set is binary, so there is no intensive margin: intervention deterministically switches the allocation from (4,0) to (2,2) or vice versa. I collect one observation for each combination (a) of source of inequality and earnings allocation (Merit×(4,0), Merit×(2,2), Luck×(4,0), Luck×(2,2)) through the strategy method, and each reviewer (i) is exposed to both treatment conditions (human and AI spectator), for eight observations per reviewer.
Secondary outcomes
Realized-oversight measures. The strategy method exposes every reviewer to the four criterion–allocation combinations with uniform frequency, whereas realized spectator decisions are not distributed uniformly. Reweighting each reviewer's eight responses by the empirical frequency of each cell among the realized decisions of spectators of the corresponding type, I report by spectator type: the implied intervention rate, the average fee paid per reviewer, the share of final allocations that differ from the reviewer's fairness ideal, and the share of worker pairs left with unequal earnings. Fairness view. Each reviewer is classified as egalitarian, meritocratic, libertarian, or other from the two decisions she made as a spectator. Spectator redistribution decisions, by spectator type and source of inequality.

Cited by

No citation data available for this registration yet.

Keywords