LLMs and e-mail marketing

How to cite

Dube, Jean-Pierre; X, Ningyin. 2023. "LLMs and e-mail marketing." AEA RCT Registry. AEARCTR-0012739. https://doi.org/10.1257/rct.12739-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jean-Pierre Dube* , Ningyin X

* Primary investigator

Registry ID
AEARCTR-0012739
DOI
https://doi.org/10.1257/rct.12739-1.0
First registered
Last updated
Status
In development
Trial dates
2023-12-31 – 2024-01-29
Countries
United States of America (national)
JEL codes
M3
Secondary IDs
L00, C9
Registry record
http://www.socialscienceregistry.org/trials/12739

Abstract

We aim to test the effectiveness of large language models (LLM) in producing “newsletter creative” (NC) compared to human writers. We collaborate with Wine Access (WA), an online wine retailer, to run a randomized controlled experiment (RCT) in a 2-week period so respondents will get/not get NC generated from different sources:
1. Control group (size = 500): The respondent receives no WA newsletter at all.
2. Test 1 (size = 9,000): The respondent receives daily WA newsletters created by the human writer team.
3. Test 2 (size = 9,000): The respondent receives daily WA newsletters created by the LLM.
4. Test 3 (size = 9,000): The respondent receives daily WA newsletters created by the human team with access to and can tune the LLM.
At the start of the experiment, each subject will be randomly assigned to one of the experimental cells above and will remain in that cell throughout the 2-week duration of the experiment. Each cell will have a human fact-checker that’s independent of the human writer team so the newsletter has accurate information. The effectiveness of NC can be assessed based on comparisons of multiple outcome variables such as site visits, revenues, and profits.

Intervention

The randomized controlled experiment (RCT) will run for a 2-week period with the following cells in parallel:
1. Control group (size = 500): The respondent receives no WA newsletter at all.
2. Test 1 (size = 9,000): The respondent receives daily WA newsletters created by the human writer team.
3. Test 2 (size = 9,000): The respondent receives daily WA newsletters created by the LLM.
4. Test 3 (size = 9,000): The respondent receives daily WA newsletters created by the human team with access to and can tune the LLM.

Experimental design

We will randomly assign approximately 27,000 customers to 4 experimental cells:
1. no intervention (control)
2. base intervention (creative designed by human team)
3. LLM Intervention (creative designed by LLM)
4. hybrid intervention (creative designed by human team after receiving creative from the LLM)

Design details

Randomization method
random number generator
Randomization unit
individual customer identification number
Clusters
N/A
Observations
approximately 27,000 customers
Arms
500 in control 9,000 in each of the intervention arms
Primary outcomes
Purchases and profits

Cited by

No citation data available for this registration yet.

Keywords