Optimising Signals in Human-AI Interaction: Experiment II

How to cite

Hoong, Ruru (Juan Ru); Dreyfuss, Bnaya. 2025. "Optimising Signals in Human-AI Interaction: Experiment II." AEA RCT Registry. AEARCTR-0015758. https://doi.org/10.1257/rct.15758-3.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Ruru (Juan Ru) Hoong* , Bnaya Dreyfuss

* Primary investigator

Registry ID
AEARCTR-0015758
DOI
https://doi.org/10.1257/rct.15758-3.0
First registered
Last updated
Status
On going
Trial dates
2025-04-14 – 2025-06-14
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/15758

Abstract

Artificial intelligence (AI) has caught pace with — and in some contexts even surpassed — humans in the ability to make predictions from data, purporting to improve decision-making. However, in cases where humans are still responsible for the final decision, biases in probabilistic reasoning can render even informative AI predictions detrimental to decision-making outcomes.

Using a randomised experiment with loan underwriters, we showed previously (#13716 in registry) that the provision of optimised AI signals can improve overall decision-making in spite of information loss. In this experiment, we explore the optimal AI policies further by adapting interventions to individual loan underwriters based on their historical decisions.

Intervention

Participants see different types of AI signals to aid their decision-making, personalised based on their historical decisions.

Experimental design

We recruit 200-300 loan underwriters/officers/processors to make loan application decisions in two stages.

Our randomisation happens int the second stage, where we personalise interventions to each underwriter for half of the underwriters, and compare it to our control benchmarks.

Design details

Randomization method
Randomisation done through Otree programme
Randomization unit
Individual
Clusters
200-300 loan officers/processors/underwriters.
Observations
We will aim to recruit 200-300 loan officers/processors/underwriters, unless otherwise limited by our recruitment. We will filter out subjects in our analysis based on attention and time checks, as well as Stage I performance as specified by our PAP.
Arms
100-150 in treatment, 50-75 in each control.
Primary outcomes
The main dependent variable we will look at is decision accuracy in State 2 (i.e. whether the decision to approve/deny the loan was ex-post correct).

Cited by

No citation data available for this registration yet.

Keywords