Human learning about AI performance

How to cite

Raux, Raphael. 2023. "Human learning about AI performance." AEA RCT Registry. AEARCTR-0012622. https://doi.org/10.1257/rct.12622-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Raphael Raux*

* Primary investigator

Registry ID
AEARCTR-0012622
DOI
https://doi.org/10.1257/rct.12622-1.0
First registered
Last updated
Status
In development
Trial dates
2023-11-29 – 2023-12-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/12622

Abstract

We study human perceptions of AI performance in mathematics and what they learn from observed AI performance. We compare prior beliefs and updating patterns between humans and AI, and test predictions of a theoretical model of performance anthropomorphism.

Intervention

We develop a survey experiment where Prolific subjects are presented with multiple-choice math questions (taken from different standardized tests). They are incentivized both to solve them and to predict the performance (likelihood of answering correctly) of someone else, either a human or an AI (ChatGPT). Incentives for beliefs will be done using the binarized scoring rule.

Experimental design

The survey is composed of 3 parts. The first part provides instructions and familiarizes participants with those they will be asked to predict performance of: humans (Prolific subjects who took the test) or ChatGPT.
The second part presents randomly selected questions (around 10) and asks participants to solve them and predict performance of humans/ChatGPT.
The last part tests the cross inference prediction: elicit prior beliefs, reveal performance on one question, and elicit posterior belief (4 possible conditions varying revealed performance).
The survey will end with basic demographic questions, as well as a question asking about prior AI familiarity.

Design details

Randomization method
Randomization is done at the Qualtrics block level between the main treatments (human/AI) and for the cross inference blocks (4 blocks). Then questions shown for the second survey part and randomly sampled from the full pool of questions: we will ensure that the same subject sees randomly sampled questions spanning the full gradient of difficulty. For cross inference, we create pools of easy/hard questions, on which ChatGPT failed/succeeded (performance is for model GPT 3.5, August 3 version). From these pools, a hard question is randomly drawn to elicit prior and posterior, and another question is drawn from one of the 4 other pools (easy fail, hard fail, easy success, hard success) to reveal performance. Random draw is done through embedded data and conditional question display.
Randomization unit
Individual survey respondents for the main human/AI treatment. Then, within each group, subjects are randomly assigned to one of the 4 possible cross inference conditions.
Clusters
2 main groups of subjects (human/AI condition), each divided into 4 sub-groups (cross inference condition).
Observations
Around 1000-1100 total subjects, excluding those failing attention checks or comprehension questions (slightly more subjects will be recruited to end up with this number).
Arms
Around 800 for AI treatment, then divided into 4 groups of around 200 for each cross inference condition. Same for the human treatment, but with around 250-300, divided into 4 groups of 62-75 subjects.
Primary outcomes
(Incentivized) beliefs in performance: both prior beliefs and posterior beliefs, following a one-shot success/mistake on a revealed question. We are interested in comparing the shape of prior and posterior beliefs between humans and AI. Our model makes two main predictions: (i) beliefs in both human and AI performance are decreasing in the (human) difficulty of the task considered; (ii) the posterior belief in both human and AI performance on a "hard" task is lower after revealing a mistake on an "easy" task, compared to the same posterior after revealing a mistake on a (different) "hard" task; conversely, posterior belief on a "hard" task is higher after revealing a success on a (different) "hard" task, compared to revealing a success on an "easy" task. We call this prediction "cross inference".

Cited by

No citation data available for this registration yet.

Keywords