Several studies have examined the uses and impact of generative AI applications aimed at improving the productivity of people in the workplace and other settings. However, little is known about the contexts in which people may be more or less willing to seek and adopt AI feedback. This study is aimed at determining how perceptions about AI affects its usage by people. We randomly vary the way in which an AI feature is introduced to users via text-based “nudges,” and then measure how users respond in terms of their willingness to try out the feature.
Intervention
For this field experiment, we are working with a company which has developed a feature that provides AI-generated feedback to users who are typing an email message. This tool gives feedback to users on attributes of their language, such as tone, reader sentiment, and clarity. Users can then choose to incorporate such feedback when communicating with others via email. The company’s product team has developed an initial text-based nudge to users who are typing an email in the form of brief text within their email window that encourages them to use the AI feature. We propose a randomized roll-out of different text-based variants of this nudge within the email product such that users in our experiment will either receive no nudge or receive one variant of a text-based nudge describing the capability of the AI feature while they are writing emails. We aim to measure the types of text-based nudges that lead to greater usage of the AI feature.
Experimental design
Information on the experimental design is hidden until the end of the trial.
Design details
Randomization method
Randomization is done via code by the company's experimentation platform.
Randomization unit
Randomization is at the level of the individual user.
Clusters
N/A.
Observations
The nudge will be rolled out to worldwide users of the company who are eligible for the AI tool. There are approximately 1 million worldwide eligible enterprise users.
Additionally, our firm partner is also providing credits that will allow their consumer users to access the AI tool in a limited way (i.e. consumers receive a specific number of credits per method to access the AI coaching tool in addition to a suite of other AI tools). There are approximately 17 million consumer users, who will be eligible for our randomized nudges for an additional period of approximately one month (Jan 16, 2025 to Feb 17, 2025) if they choose to use the email product during this period.
Arms
Users are evenly distributed into each of the seven randomized arms with approximately 145,000 enterprise users in each group who are eligible to receive the nudges. Users in the consumer sample will also be evenly distributed into each of the seven randomized arms.
Note that the number of users who may receive the nudges is likely smaller than those eligible. Nudges are triggered for users based on specific criteria determined by the product’s code. Based on an initial test of the nudge system, approximately 10,000 nudges were triggered in a two-week period for 30,000 users. Assuming nudges are triggered for users at a similar rate, this means that we can expect to see at least 48,000 nudges in a two-week period per treatment group that is assigned a nudge. (Note that nudges may be triggered several times for the same user.)
We plan to examine outcomes for all users eligible for the nudge as well as the subset who ended up receiving nudges.
Primary outcomes
The primary outcome of interest is whether our randomized nudges influence users’ decisions about using the AI feature.
Secondary outcomes
As a secondary outcome, we will measure whether people retain the AI-generated feedback provided by the AI tool, as measured by users pressing the "apply suggestions" button after trying the tool.
Cited by
No citation data available for this registration yet.