Evaluating User Bias in Content Preference: AI-Generated Texts Versus Popular Online Platforms

How to cite

Nesseler, Cornel; Paklina, Sofia; Parshakov, Petr; Naidenova, Iuliia; Matkin, Nikita. 2024. "Evaluating User Bias in Content Preference: AI-Generated Texts Versus Popular Online Platforms." AEA RCT Registry. AEARCTR-0014231. https://doi.org/10.1257/rct.14231-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Cornel Nesseler* , Sofia Paklina , Petr Parshakov , Iuliia Naidenova , Nikita Matkin

* Primary investigator

Registry ID
AEARCTR-0014231
DOI
https://doi.org/10.1257/rct.14231-1.0
First registered
Last updated
Status
In development
Trial dates
2024-10-01 – 2025-01-01
Countries
Private
JEL codes
C80, D83, C9
Registry record
http://www.socialscienceregistry.org/trials/14231

Abstract

This study explores user preferences between AI-generated explanations and those curated by popular platforms like Quora. We identify popular questions in five scientific fields and use responses from platforms and AI. Through a field experiment, participants were presented with texts from both sources and asked to indicate their preferred explanation. The findings have the potential to reveal trends in user bias. Our analysis provides insights into the evolving role of AI in content generation and its comparison with traditional human-curated knowledge platforms.

Experimental design

We recruit students at a higher education institute and participants from a popular survey platform.

All participants are first presented with an overview regarding the aim of the experiment, information regarding who to contact, and information regarding how the data will be stored. The experiment will gather no data that can reveal individuals.

We first ask demographic questions (age and gender) and the educational background. Participants are then presented with five questions from five different scientific fields (depending on the education they selected). Each question includes two responses. One response is generated through AI. The other is a human generated response from popular platforms. The participants have to state which response they prefer (the response options are binary: Text 1 or Text 2). We do not reveal if a response is AI or human generated.

Design details

Randomization method
We use a statistical application, R, Stata, or Mathematica. The randomization uses a code that makes it reproducible.
Randomization unit
We randomize at the survey level. That means that we construct different surveys, with a different structure.
Clusters
We plan to include university participants and participants from popular surveys (similar to MTurk).
Observations
We plan to have at least 500 participants. This results in 5,000 observations as each participant responds to ten questions.
Arms
2 groups (students and participants from popular surveys), 500 participants (in total) from both groups, each responding to ten questions.
Primary outcomes
The primary outcome is the evaluation of the text.
Secondary outcomes
Demographic information: Gender and age Education

Cited by

No citation data available for this registration yet.

Keywords