Towards a Cheap Personalization of Political Messaging with Large Language Models?

How to cite

Widmer, Philine. 2024. "Towards a Cheap Personalization of Political Messaging with Large Language Models?." AEA RCT Registry. AEARCTR-0014357. https://doi.org/10.1257/rct.14357-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Philine Widmer*

* Primary investigator

Registry ID
AEARCTR-0014357
DOI
https://doi.org/10.1257/rct.14357-1.1
First registered
Last updated
Status
In development
Trial dates
2024-09-16 – 2025-09-30
Countries
Private
JEL codes
L82, C53, D72, O33, Z13
Registry record
http://www.socialscienceregistry.org/trials/14357

Abstract

Large languge models (LLMs, like ChatGPT) have been shown to be persuasive in political messaging. Less is known about whether there are personalization gains when using LLMs for political messaging (i.e., when the LLM is asked to tailor a message to a specific demographic group). This project investigates if personalized messages are more effective than unpersonalized messages.

Experimental design

We first randomly assign each participant to one out of six policy topic-stances (three topics with a pro/contra statement each). Afterward, each participant is shown one non-personalized, partly personalized, or fully personalized political message on the assign topic-stance. We then evaluate whether (more) personalized messages are more persuasive.

Design details

Randomization method
Randomization within survey (by a computer)
Randomization unit
Individual
Clusters
5000 individuals
Observations
5000 individuals
Arms
Treatment is not binary. See the intervention for an explanation. For example, for a participant who identifies as a Democrat or Republican, the message they are shown will match their partisan affiliation with a 50% probability. Similarly, in terms of age, the probability of the message being matched is 33% (considering that the categories are young, middle-aged, old). Since the message is randomly drawn from all possible messages, all of the covariates being matched is rare: 50% (ideology with Liberal/Conservative) * 50% (political partisanship with Democrat/Republican) * 20% (race with Black, White, Hispanic, Asian, Native American) * 50% (gender with male/female) * 33% (financial status with poor, middle-class, upper-class) * 33% (age with young, middle-aged, old) = 0.28%. For categorical variables, note that, in practice, some participants will likely be part of a category not considered in our prompts (e.g., those whose partisanship is "Independent"). Accordingly, the share of those with a match for the categorical variables is, in expectation, slightly lower than what the calculation above suggests. For the continuous variables, we will follow our implementation partner's mapping into categories and evaluate its robustness to other cutoffs.
Primary outcomes
We ask participants for their stance on the policy at hand, for their agreement with the arguments in the message, for the importance they attribute to the topic, how much the message resonates with their political opinion, how much they identify with the message, and how easy or hard the message was to read. Furthermore, we give them the option to generate donations for or against the policy proposed in the message by solving up to ten counting quizzes (the quizzes are voluntary and do not affect the participants' own remuneration).
Secondary outcomes
For each message, we make the LLM explain its "reasoning" variable-by-variable. That is, for every variable (e.g., age), the LLM explains why the message should be appealing to somebody who is young (if the message was tailored to a young person). At the end of the survey, participants choose which aspect (if any) they find most or least convincing. Furthermore, we ask participants at the end of the survey whether they think that an artificial intelligence, a human, or both of them authored the post together.

Cited by

No citation data available for this registration yet.

Keywords