Humans, Artificial Intelligence and Text-based Misinformation
How to cite
Hazra, Sanchaita; Dugar, Subhasish; Bhattacharya, Haimanti; Majumder, Bodhisattwa. 2023. "Humans, Artificial Intelligence and Text-based Misinformation." AEA RCT Registry. AEARCTR-0012535. https://doi.org/10.1257/rct.12535-2.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Text-based misinformation is pervasive, yet evidence is scarce regarding people’s ability to differentiate truth from deceptive content in textual form. We conduct a laboratory experiment utilizing data from a TV game show, where natural conversations surrounding an underlying objective truth between individuals with conflicting objectives lead to intentional deception. Initially, we elicit participants’ guesses about the underlying truth by exposing them to transcribed conversations from random episodes. Borrowing tools from computing, we demonstrate that certain AI algorithms exhibit comparable truth detection performance to humans, despite the algorithms relying solely on language cues while humans have access to language and audio-visual cues. Our model identifies accurate language cues not always detected by humans, suggesting the potential for collaborative efforts between humans and algorithms to enhance truth detection abilities. Our research takes an interdisciplinary approach and aims to ascertain whether human-AI teams can outperform individual humans in spotting the truth amid misinformation appearing in textual form. Subsequently, we pursue several lines of inquiry: Do individuals seek the assistance of an artificial intelligence (AI) tool to aid their discernment of truth from text-based misinformation? Are individuals willing to pay for the service provided by the AI? We also investigate factors that may influence individuals’ reluctance in or excessive dependence on seeking AI assistance, such as “AI aversion” or its absence, as well as overconfidence in one’s ability to identify the truth. Furthermore, we examine, while controlling for the predictive accuracies of both the majority of humans and the AI tool, whether individuals, in comparison to the AI tool, are more or less inclined to submit the same guess that a majority of other individuals had submitted for that episode as their own. Lastly, we examine potential gender differences concerning these questions.
Intervention
Despite the proliferation of text-based misinformation on social media, there is a gap in our understanding of individuals' ability to detect deception in textual form. We use a transcribed version of a novel TV gameshow in which the conversations mimic a social media platform where a third party seeks to ascertain the actual truth concerning a particular topic (e.g., the accuracy of economic data or historical events) by observing online discussions among individuals who have conflicting motives related to that topic, and where there is an underlying objective truth. We ask the subjects to discern and guess the truth from these transcripts. Subsequently, we study the willingness of the subjects to switch to a guess made by an artificial intelligent (AI) system. We vary the accuracy of this AI system and aim to study how this has an effect on the switching behavior of individuals. Furthermore, we seek if subjects are willing to pay for this service provided by an AI.
Experimental design
We conduct this project in two phases. The first phase focuses on building the AI system to detect false information with the help of ChatGPT. In the second phase of the project, we will utilize the methods of the economics experiments. We invite individual human subjects, who will be presented with the transcripts from one of the randomly chosen sessions of the game show. The subject will be entrusted with identifying the truth and will be paid a fixed sum of monetary rewards for correctly identifying, tantamount to detecting deception or sorting out misinformation.
Design details
Cited by
No citation data available for this registration yet.