The Buy-In Effect: When Increasing Initial Effort Motivates Behavioral Follow-Through

How to cite

Dykstra, Holly; Whillans, Ashley; Flaherty, Shibeal O'. 2024. "The Buy-In Effect: When Increasing Initial Effort Motivates Behavioral Follow-Through." AEA RCT Registry. AEARCTR-0013133. https://doi.org/10.1257/rct.13133-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Holly Dykstra* , Ashley Whillans , Shibeal O' Flaherty

* Primary investigator

Registry ID
AEARCTR-0013133
DOI
https://doi.org/10.1257/rct.13133-1.0
First registered
Last updated
Status
Completed
Trial dates
2019-06-29 – 2020-02-29
Countries
United States of America (Oregon)
Registry record
http://www.socialscienceregistry.org/trials/13133

Abstract

Behavioral interventions often focus on reducing friction to encourage behavior change.

The intention for our study is to provide evidence for a 'Buy-In Effect.' We test whether adding friction upfront in a sign-up process encourages buy-in and behavioral follow-through among employees who sign-up for a carpool program. Research in behavioral economics has shown that automatically enrolling people into programs can have positive effects on outcomes (e.g., Beshears, Choi, Laibson, & Madrian, 2010). However, these studies have largely shown benefits in settings where the outcome is a natural and automatic consequence of the policy: for example, in the automatic enrollment work on retirement savings, the outcome policymakers are looking for is larger retirement savings, which happens automatically when people are opted into these accounts (Madrian and Shea, 2001).

In this project, however, we study a program whereby people have to use deliberate effort to continuously engage with the service after initial enrollment. Policymakers in this setting therefore hope that participants engage with the carpool program and actively use the tool after they are initially registered, but this decision requires continuous active and engaged effort on behalf of participants themselves.

In this setting, we hypothesize that there might be a 'buy-in effect', where people are more engaged with the tool when they’ve made the active choice to opt in, even if they would have opted in anyway in the absence of automatic enrollment. This may mean that it would be more effective to actively recruit people to take up the tool rather than having them passively enrolled, and would have important consequences for behaviorally informed interventions in similar contexts.

We collaborate with the Oregon Department of Transportation (ODOT) to test whether adding friction during an initial sign-up process for a new carpool platform increases usage and follow-through. We design two versions of an email being sent to previously inactive users of ODOT's old carpool platform, with the first version ('More Effort') involving additional friction in the sign-up process in the form of more steps to create one's account on the new carpool platform, and the second version ('Less Effort') simply involving individuals having to create a new password in order to access their account on the new carpool platform.

To evaluate the impact of the intervention on participant follow-through, we conduct a randomized controlled trial (RCT). We test for evidence of the buy-in effect by measuring three outcomes related to participants' follow-through: (1) number of carpool platform sign-ups, (2) number of trips logged on the carpool platform, and (3) number of miles logged on the carpool platform.

Intervention

Our RCT investigates evidence for a 'Buy-In Effect', whereby we test whether adding friction to carpool platform sign-up can increase follow-through in the form of sign-ups to the platform and subsequent usage of the platform. We partner with the Oregon Department of Transportation (ODOT) to conduct the field experiment. The intervention involves two versions of a sign-up process and corresponding email ('Less Effort' vs. 'More Effort') that is sent to participants and that encourages existing users of the old carpool platform, 'Drive Less Connect' to sign up for a new carpool platform, 'Get There Oregon'. In the 'Less Effort' email condition, participants only have to complete one step to access the new carpooling platform: resetting their password. In the 'More Effort' email condition, participants have to recreate their account to sign up to the platform, which involves inputting additional information in addition to creating a new password.

Experimental design

We conduct a randomized field experiment in which participants are randomly assigned to a 'More Effort' or 'Less Effort' group.

Design details

Randomization method
- Randomization is conducted by assigning each individual a randomized integer, and then assigning treatment group based on a cutoff number. - Randomization is done with a computer in an office.
Randomization unit
Individual randomization.
Clusters
None.
Observations
26,000-27,500 individuals. We note that the number of observations specified in our IRB application was 40,000, based on an estimate provided by ODOT; however, the actual sample size available in the ODOT database that is feasible to include based on our inclusion criteria (inactive user - have not logged in or used the old carpool platform for 6 months or more) is lower at N = 27,227.
Arms
Individuals are randomly assigned into one of two experimental groups: 'More Effort' group: 13,000-13,750 individuals 'Less Effort' group: 13,000-13,750 individuals The emails are targeted at previous users of ODOT's old carpool platform. Thus, in the IRB application, it is specified that the population under study would be individuals who were users of the previous app and elected to receive notifications from the ODOT. However, during the initial phase of the evaluation, we learned that ODOT wanted to target users of the old carpool platform who they considered 'inactive' - those who had not logged into the old carpool platform in six or more months prior to launching our experiment. Therefore, we conducted randomization on the individuals on ODOT's database who had not been active for six or more months prior to the trial (87% of all their users).
Primary outcomes
The primary outcomes are specified in the IRB application (granted on June 26, 2019).* Specifically, we investigate three outcomes that we hypothesize are likely to be impacted by the more effortful sign-up process: (1) number of sign-ups on the carpool platform (2) number of trips logged on the carpool platform (3) number of miles logged on the carpool platform * The IRB application specifies that we will measure: (1) whether the user logs into the new tool at all (number of account sign-ups) (2) whether the user looks for carpool matches (3) whether the user logs trips in the new tool (number of trips logged) For (2), we discovered that it was not possible for ODOT to provide data on whether the user searches for carpool matches. In addition, ODOT informed us that they could provide us with 'number of miles logged' on the carpool platform. This indicator is a good measure of later follow-through; therefore, we substitute (2) with this outcome. We also conduct balance checks between the two treatment conditions with respect to gender, race, and whether the participant receives an email in June or October 2019. These analyses are not pre-specified in the IRB application. We also have data from ODOT on participants' home and work zip codes, not pre-specified in the IRB application. This does not comprise the primary analysis, but provides us with the ability to conduct additional robustness checks.
Partners
Name: Oregon Department of Transportation Partner_type: government Url: https://www.oregon.gov/odot/pages/index.aspx

Papers from this trial

  1. Dykstra, H., O' Flaherty, S., & Whillans, A. (2023). The Buy-In Effect: When Increasing Initial Effort Motivates Behavioral Follow-Through. Harvard Business School Working Paper 24-020. https://www.hbs.edu/ris/Publication%20Files/24-020_a7c183d8-afcc-4e9e-a764-43cb131edf60.pdf

Cited by

No citation data available for this registration yet.

Keywords