Do people trust generative AI, and is it trustworthy? An experiment with ChatGPT

How to cite

Livingston, Jeffrey; Shen, Samson; Rankich, Kobe. 2024. "Do people trust generative AI, and is it trustworthy? An experiment with ChatGPT." AEA RCT Registry. AEARCTR-0013792. https://doi.org/10.1257/rct.13792-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jeffrey Livingston* , Samson Shen , Kobe Rankich

* Primary investigator

Registry ID
AEARCTR-0013792
DOI
https://doi.org/10.1257/rct.13792-1.0
First registered
Last updated
Status
In development
Trial dates
2024-07-15 – 2024-08-30
Countries
United States of America ()
JEL codes
C91, C93, D90, D91
Secondary IDs
n/a
Registry record
http://www.socialscienceregistry.org/trials/13792

Abstract

We measure whether and how much people trust generative AI by having them play Trust Games (Berg et al. 1995) with ChatGPT. Subjects are given $10 and are can send any portion of this to ChatGPT, which is tripled. ChatGPT then decides how much of what it receives to the subject. There are three treatments: Treatment Self-Interest, Treatment Fairness, and Treatment No Prompt.

In each treatment, we vary the prompt that instructs ChatGPT how to play the game. In Treatment Self-Interest, ChatGPT is told to determine how much to return to the subject "as a rational person would." In Treatment Fairness, ChatGPT is told to determine how much to return to the subject "as a person who believes in fairness and reciprocity" would. In Treatment No Prompt, ChatGPT is given no instructions on how it should decide how much to return to the subject.

The experiment is within-subject; each person plays the game three times (once for each treatment). We randomize the order of the treatments. One game is randomly chosen to determine payments to the subject.

Intervention

Experiment 1:
We measure whether and how much people trust generative AI by having them play Trust Games (Berg et al. 1995) with ChatGPT. Subjects are given $10 and are can send any portion of this to ChatGPT, which is tripled. ChatGPT then decides how much of what it receives to the subject. There are three treatments: Treatment Self-Interest, Treatment Fairness, and Treatment No Prompt.

In each treatment, we vary the prompt that instructs ChatGPT how to play the game.

In Treatment Self-Interest, ChatGPT is told to determine how much to return to the subject "as a rational, self-interested person would."

In Treatment Fairness, ChatGPT is told to determine how much to return to the subject "as a person who believes in fairness and reciprocity" would.

In Treatment No Prompt, ChatGPT is given no instructions on how it should decide how much to return to the subject.

The experiment is within-subject; each person plays the game three times (once for each treatment). We randomize the order of the treatments. One game is randomly chosen to determine payments to the subject.

The full experiment interface and all instructions can be viewed in the document attached in the Docs & Materials section.

Experiment 2:
With the same set of treatments, we run the same trust game where we send ChatGPT $10 in every play and record how much ChatGPT decides to return. We do this with several different versions of ChatGPT:
1. ChatGPT 3.5
2. ChatGPT 4
3. ChatGPT 4o
4-6: The same versions, but through the ChatGPT API instead of the main interface at chat.openai.com.

Experimental design

Experiment 1:
We measure whether and how much people trust generative AI by having them play Trust Games (Berg et al. 1995) with ChatGPT. The subject is the first mover. They are given $10 and are can send any portion of this to ChatGPT, which is tripled. ChatGPT is the second mover. It then decides how much of what it receives to the subject. There are three treatments: Treatment Rational, Treatment Fairness, and Treatment No Prompt. The experiment is within-subject; each person plays the game three times (once for each treatment). We randomize the order of the treatments. One game is randomly chosen to determine payments to the subject.

In each treatment, we vary the prompt that instructs ChatGPT how to play the game.

In Treatment Rational, ChatGPT is told to determine how much to return to the subject "as a rational person would." The prompt that is entered to get ChatGPT to play the game reads as follows:
"Play a game with me where I am given $10. I will first tell you how much of this $10 I want to send to you. Whatever I send is tripled, so if I send you $10, you receive $30 and I have $0. If I send $5 to you, you receive $15 and I have $5. You can then choose to return any amount of what you receive back to me, including keeping all of the money for yourself, but this amount is not tripled. We will play the game one time. I will start by telling you how much I wish to send to you. At the end, state how much money each of us end up with. My payment should be the amount of money that I keep instead of sending it to you, plus the amount that you send back to me. When deciding how much to return to me, you should play as a rational, self-interested person would."

In Treatment Fairness, ChatGPT is told to determine how much to return to the subject "as a person who believes in fairness and reciprocity" would. The prompt that is entered to get ChatGPT to play the game reads as follows:
"Play a game with me where I am given $10. I will first tell you how much of this $10 I want to send to you. Whatever I send is tripled, so if I send you $10, you receive $30 and I have $0. If I send $5 to you, you receive $15 and I have $5. You can then choose to return any amount of what you receive back to me, including keeping all of the money for yourself, but this amount is not tripled. We will play the game one time. I will start by telling you how much I wish to send to you. At the end, state how much money each of us end up with. My payment should be the amount of money that I keep instead of sending it to you, plus the amount that you send back to me. When deciding how much to return to me, you should play as a person who believes in fairness and reciprocity would."

In Treatment No Prompt, ChatGPT is given no instructions on how it should decide how much to return to the subject. The prompt that is entered to get ChatGPT to play the game reads as follows:
"Play a game with me where I am given $10. I will first tell you how much of this $10 I want to send to you. Whatever I send is tripled, so if I send you $10, you receive $30 and I have $0. If I send $5 to you, you receive $15 and I have $5. You can then choose to return any amount of what you receive back to me, including keeping all of the money for yourself, but this amount is not tripled. We will play the game one time. I will start by telling you how much I wish to send to you. At the end, state how much money each of us end up with. My payment should be the amount of money that I keep instead of sending it to you, plus the amount that you send back to me."

After completing the three plays of the game, the subject completes a 16 question survey which asks questions about how trusting the person is, demographic information, the subject's previous experiences with ChatGPT, and their expectations for the impacts ChatGPT will have in the future The survey questions can be viewed in the document attached in the Docs & Materials section.

This document also displays the full experiment interface.

Experiment 2:
With the same set of treatments (No Prompt, Rational, and Fairness & Reciprocity), we run the same trust game where we send ChatGPT $10 in every play and record how much ChatGPT decides to record. We do this with several different versions of ChatGPT:
1. ChatGPT 3.5
2. ChatGPT 4
3. ChatGPT 4o
4. ChatGPT 3.5 API version
5. ChatGPT 4 API version
6. ChatGPT 4o API version

Design details

Randomization method
The order in which the subjects do each treatment is determined via simple randomization. Each subject's order is determined randomly by computer and independently for each subject.
Randomization unit
Individual subject.
Clusters
n/a
Observations
Experiment 1: 360 Experiment 2: 1800
Arms
Experiment 1: 120 per treatment order, though since we are dong a simple randomization, there may be a few more or a few less than 120 assigned to each treatment order. Experiment 2: 100 per treatment condition (300 total) for each of the 6 versions of ChatGPT, so 1800 total.
Primary outcomes
Experiment 1: 1. The amount of money (out of $10) that the subject decides to send to ChatGPT in the first move of the trust game. Experiment 2: 1. The amount that ChatGPT decides to return to the subject, (raw amount out of $30).
Secondary outcomes
Experiment 1: 1. The amount that ChatGPT decides to return to the subject, both the raw amount and as a percentage of the amount received. 2.

Cited by

No citation data available for this registration yet.

Keywords