LLMs and e-mail marketing Follow-up Study II
How to cite
Xu, Ningyin; Dubé, Jean-Pierre. 2025. "LLMs and e-mail marketing Follow-up Study II." AEA RCT Registry. AEARCTR-0015519. https://doi.org/10.1257/rct.15519-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We aim to test the effectiveness of large language models (LLM) in producing “newsletter creative” (NC) compared to human writers. We collaborate with Wine Access (WA), an online wine retailer, to run a randomized controlled experiment (RCT) in a 14-day period so respondents will get/not get NC generated from different sources:
1. Test 1 (size = 9,000): The respondent receives daily WA newsletters created by the human writer team.
2. Test 2 (size = 9,000): The respondent receives daily WA newsletters created by the LLM.
3. Test 3 (size = 9,000): The respondent receives daily WA newsletters created by the human marketing team with access to and can tune the LLM.
At the start of the experiment, each subject will be randomly assigned to one of the experimental cells above and will remain in that cell throughout the 14-day duration of the experiment. Each cell will have a human fact-checker that’s independent of the human writer team so the newsletter has accurate information. The effectiveness of NC can be assessed based on comparisons of multiple outcome variables such as site visits, revenues, and profits.
Intervention
The randomized controlled experiment (RCT) will run for a 14-day period with the following cells in parallel:
1. Test 1 (size = 9,000): The respondent receives daily WA newsletters created by the human writer team.
2. Test 2 (size = 9,000): The respondent receives daily WA newsletters created by the LLM.
3. Test 3 (size = 9,000): The respondent receives daily WA newsletters created by the human marketing team with access to and can tune the LLM.
Experimental design
We will randomly assign approximately 27,000 customers to 3 experimental cells:
1. base intervention (creative designed by human team)
2. LLM Intervention (creative designed by LLM)
3. hybrid intervention (creative designed by human marketing team after receiving creative from the LLM)
Design details
Cited by
No citation data available for this registration yet.