LLMs and e-mail marketing Follow-up Study II

How to cite

Xu, Ningyin; Dubé, Jean-Pierre. 2025. "LLMs and e-mail marketing Follow-up Study II." AEA RCT Registry. AEARCTR-0015519. https://doi.org/10.1257/rct.15519-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Ningyin Xu* , Jean-Pierre Dubé

* Primary investigator

Registry ID
AEARCTR-0015519
DOI
https://doi.org/10.1257/rct.15519-1.0
First registered
Last updated
Status
On going
Trial dates
2025-02-24 – 2025-03-09
Countries
Private
JEL codes
M3
Secondary IDs
L00, C9
Registry record
http://www.socialscienceregistry.org/trials/15519

Abstract

We aim to test the effectiveness of large language models (LLM) in producing “newsletter creative” (NC) compared to human writers. We collaborate with Wine Access (WA), an online wine retailer, to run a randomized controlled experiment (RCT) in a 14-day period so respondents will get/not get NC generated from different sources:
1. Test 1 (size = 9,000): The respondent receives daily WA newsletters created by the human writer team.
2. Test 2 (size = 9,000): The respondent receives daily WA newsletters created by the LLM.
3. Test 3 (size = 9,000): The respondent receives daily WA newsletters created by the human marketing team with access to and can tune the LLM.
At the start of the experiment, each subject will be randomly assigned to one of the experimental cells above and will remain in that cell throughout the 14-day duration of the experiment. Each cell will have a human fact-checker that’s independent of the human writer team so the newsletter has accurate information. The effectiveness of NC can be assessed based on comparisons of multiple outcome variables such as site visits, revenues, and profits.

Intervention

The randomized controlled experiment (RCT) will run for a 14-day period with the following cells in parallel:
1. Test 1 (size = 9,000): The respondent receives daily WA newsletters created by the human writer team.
2. Test 2 (size = 9,000): The respondent receives daily WA newsletters created by the LLM.
3. Test 3 (size = 9,000): The respondent receives daily WA newsletters created by the human marketing team with access to and can tune the LLM.

Experimental design

We will randomly assign approximately 27,000 customers to 3 experimental cells:
1. base intervention (creative designed by human team)
2. LLM Intervention (creative designed by LLM)
3. hybrid intervention (creative designed by human marketing team after receiving creative from the LLM)

Design details

Randomization method
random number generator
Randomization unit
individual customer identification number
Clusters
N/A
Observations
approximately 27,000 customers
Arms
9000 in each intervention arms
Primary outcomes
Purchases and profits

Cited by

No citation data available for this registration yet.

Keywords