LLMs and e-mail marketing
How to cite
Dube, Jean-Pierre; X, Ningyin. 2023. "LLMs and e-mail marketing." AEA RCT Registry. AEARCTR-0012739. https://doi.org/10.1257/rct.12739-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We aim to test the effectiveness of large language models (LLM) in producing “newsletter creative” (NC) compared to human writers. We collaborate with Wine Access (WA), an online wine retailer, to run a randomized controlled experiment (RCT) in a 2-week period so respondents will get/not get NC generated from different sources:
1. Control group (size = 500): The respondent receives no WA newsletter at all.
2. Test 1 (size = 9,000): The respondent receives daily WA newsletters created by the human writer team.
3. Test 2 (size = 9,000): The respondent receives daily WA newsletters created by the LLM.
4. Test 3 (size = 9,000): The respondent receives daily WA newsletters created by the human team with access to and can tune the LLM.
At the start of the experiment, each subject will be randomly assigned to one of the experimental cells above and will remain in that cell throughout the 2-week duration of the experiment. Each cell will have a human fact-checker that’s independent of the human writer team so the newsletter has accurate information. The effectiveness of NC can be assessed based on comparisons of multiple outcome variables such as site visits, revenues, and profits.
Intervention
The randomized controlled experiment (RCT) will run for a 2-week period with the following cells in parallel:
1. Control group (size = 500): The respondent receives no WA newsletter at all.
2. Test 1 (size = 9,000): The respondent receives daily WA newsletters created by the human writer team.
3. Test 2 (size = 9,000): The respondent receives daily WA newsletters created by the LLM.
4. Test 3 (size = 9,000): The respondent receives daily WA newsletters created by the human team with access to and can tune the LLM.
Experimental design
We will randomly assign approximately 27,000 customers to 4 experimental cells:
1. no intervention (control)
2. base intervention (creative designed by human team)
3. LLM Intervention (creative designed by LLM)
4. hybrid intervention (creative designed by human team after receiving creative from the LLM)
Design details
Cited by
No citation data available for this registration yet.