AI, Information Processing and Dissemination Study 2

How to cite

Pethig, Florian; Bauer, Kevin; Hoehle, Hartmut. 2023. "AI, Information Processing and Dissemination Study 2." AEA RCT Registry. AEARCTR-0011541. https://doi.org/10.1257/rct.11541-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Florian Pethig* , Kevin Bauer , Hartmut Hoehle

* Primary investigator

Registry ID
AEARCTR-0011541
DOI
https://doi.org/10.1257/rct.11541-1.0
First registered
Last updated
Status
In development
Trial dates
2023-06-08 – 2023-06-30
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/11541

Abstract

Humans increasingly engage with information created or augmented by artificial intelligence (AI). While prior work mainly considers the demand for information, little work has studied how humans process and disseminate AI-generated information once they obtain it. Specifically, our research aims to contribute to this novel research stream by studying (i) how humans process and act upon AI-generated compared to human-generated information, and (ii) how humans disseminate AI-generated information. We will perform a series of controlled, incentivized online experiments. We leverage controlled incentivized experiments to circumvent endogeneity concerns that naturally arise when it comes to being exposed to information, e.g., self-selection, informational echo chambers, and (mis)trust in specific sources.

Experimental design

We ask participants about their news consumption and knowledge of Economics-related and COVID-related topics. Next, they read two articles that either contain information about an Economics-related topic or a COVID-related topic. After participants have read each article, they answer incentivized control questions about the article's content, trust in the information, and report their perception about the article’s objectivity, quality, leaning, and reliability (we use adapted version of the questions by Bachmann et al. 2022 and self-written items). Additionally, we ask participants to rate whether the information in the article is useful to answer prediction questions related to each article (e.g., how the unemployment rate will develop) and they should summarise the information of the article for a friend (dissemination). As their final task, participants have to answer several questions on their demographics, political orientation, and preferences.

The human-generated articles are curated texts from major US news outlets, where we combine different text pieces. To transform the human-generated information, we will use the Generative Pre-trained Transformer 3 (GPT-3), a state-of-the-art natural language processing (NLP) model. The AI-generated articles are paraphrasings from the original human-generated article that the GPT-3 model produces. In a between-subject fashion we vary participants’ knowledge about the source of the information. Overall, we employ 6 treatments where we either do not reveal the source (and the set of possible sources), where we correctly reveal the source, or where we incorrectly reveal the opposite source:

T1: human-generated, correct disclosure
T2: AI-generated, correct disclosure
T3: human-generated, incorrect disclosure
T4: AI-generated, incorrect disclosure
T5: human-generated, opacity (no knowledge about possible source)
T6: AI-generated, opacity (no knowledge about possible source)

In our analyses we will compare how the labelling and participants' prior knowledge of a topic affect their perceptions about the quality, leaning, and usefulness of the information. At the end of the study, we debrief participants in treatments T3 and T4 and inform them that the article they have read was actually generated by a human / AI.

Design details

Randomization method
Randomization done by survey software.
Randomization unit
Individual
Clusters
600 individuals
Observations
600 individuals
Arms
100 individuals per treatment
Primary outcomes
Please answer the following questions based on the information provided in the text you just read. You will receive $0.25 for each correctly answered question. (Each participant only had to answer questions about one economics-related and one Covid-related text.) Economics 1. US Job Report April 23 a. Which sector had the highest job gains in April? i. Retail ii. Professional and Business Services (TRUE) iii. Health Care b. According to the article, the added jobs in April ... i. beat expectations (TRUE) ii. were below expectations iii. met expectations c. According to the article, what happened to hourly wages in April? i. They remained the same ii. They decreased iii. They increased (TRUE) 2. US GDP Q1 2023 a. What was a reason behind the sharp reduction in overall growth in the first quarter? i. Higher interest rates (TRUE) ii. Higher consumer spending iii. Increased business inventories b. Which sector showed a good performance in the first quarter, according to the article? i. Housing market ii. Spending on Goods (TRUE) iii. Banking sector c. What was the reason behind the reduction in business inventories in the first quarter? i. Anticipation of a coming economic decline (TRUE) ii. Increased lending iii. Reduced interest rates COVID-19 1. Pandemic National Emergency a. Why did the White House initially oppose the congressional resolution to end the national emergency? i. It would create disorder and uncertainty for health care providers and millions of people. (TRUE) ii. It would lead to an increase in Covid 19 infections. iii. It would limit the government's ability to respond to the pandemic. b. How did most Democrats in the House vote on the congressional resolution? i. They voted in favor of it. ii. They voted against it. (TRUE) iii. They abstained from voting. c. According to the article, what was the purpose of the national emergency related to COVID-19? i. To support the country's economic, health and welfare systems. (TRUE) ii. To extend the use of emergency powers by the president. iii. To limit the entry of foreigners into the US. 2. Pan-variant vaccine a. What journal published the report on the new vaccine's results in animals? i. Frontiers in Immunology (TRUE) ii. The New England Journal of Medicine iii. Nature Medicine b. What is the role of artificial intelligence in the development of the new COVID-19 vaccine? i. To create routine booster doses for existing vaccines. ii. To teach the body to identify and hinder the spike proteins of the virus iii. To identify substances that could be effective against the virus. (TRUE) c. What is the status of clinical trials for the new COVID-19 vaccine? i. They have not yet begun. ii. They have been completed and the vaccine has been approved for use in humans. iii. They are currently ongoing. (TRUE) Attitudes toward and perceptions of the article: 1. On a scale of -3 to +3, with negative numbers representing left leaning or liberal skew, positive numbers representing right leaning or conservative skew, and 0 representing true neutral, how would you rate the article? 2. The article focuses on arguments over emotions in their coverage. 3. The article offers a comprehensive overview of all the important events. 4. The article provides objective news. 5. I generally trust the information presented in the article. 6. I generally distrust the information presented in the article. 7. The information given in the article is accurate. 8. The article is well-written 9. The article is boring. 10. I paid attention when reading the article. 11. I put effort into reading the article. 12. I felt personally involved with the issue covered by the article. 13. I thought deeply about the information contained in the article. 14. On a scale of 1 to 7, 1 being not reliable at all and 7 being very reliable, how would you rate the information in the article? 15. On a scale of 1 to 7, with 1 being not inspiring at all and 7 being extremely inspiring, how would you rate the article you just read? 16. If someone is interested in the topic of the article you just read, how likely is it that you would recommend that article to others? Prediction questions: Economics 1. US Job Report April 2023 o After having read the article, what is the probability that the US unemployment rate will remain below 4.0% in 2023? 2. US GDP Q1 2023 o After having read the article, what is the probability that the GDP will shrink (negative growth rate) in the next three quarters of 2023? COVID-19 3. Pandemic National Emergency a. After having read the article, what is the probability that the end of the Covid-19 national emergency will have a negative impact on medical care for affected people? 4. Pan-variant vaccine a. After having read the article, what is the probability that an effective Covid 19 vaccine against all current and future variants will be developed in the next two years? Imagine that you would like to inform a friend about the content of the article that you just read. Please write a concise summary that captures the most important aspects (max. 100 words).

Cited by

No citation data available for this registration yet.

Keywords