Open source is key to innovation, but we know little about how to incentivize it. In this paper, we examine the impact of a program providing monetary incentives to motivate innovators to contribute to open source. The Sponsors program was introduced by GitHub in May 2019 and enabled organizations and individuals alike to reward developers for their open source work on the platform. To study this program, we collect fine-grained data on about 100,000 GitHub users, their activities, and sponsorship events. Using a difference-in-differences approach, we document two main effects. The first is that developers who opted into the program, which does not entail receiving a financial reward, increased their output after the program’s launch. The second is that the actual receipt of sponsorship has a long-lasting negative effect on innovation, as measured by new repository creation, regardless of the amount of money received. We estimate a similar decline in other community-oriented tasks, but not in coding effort. While the program’s net effect on users’ innovative output appears to be positive, our study shows that receiving an extrinsic reward may crowd out developers’ intrinsic motivation, diverting their effort away from community and service-oriented activities on open source.
Experimental design
This is a survey that aims to understand the motivations behind GitHub users' contribution to open source. The experimental design involves a random assignment of the treatment to each individual, where the treatment involves scenarios with varying amounts and types of sponsorships provided to individuals 6 months after they enrolled in the GitHub Sponsorships program.
Design details
Randomization method
Randomization is done online by Qualtrics as it decides which experimental arm to show the user.
Randomization unit
Each individual is randomized into a treatment arm. The randomization does not depend on any of the individuals' attributes (i.e. there is no clustering done that affects the randomization).
Clusters
Our survey focuses on individuals, and they are not clustered at any level.
Observations
We have a set of emails available for over 10,000 GitHub users who enrolled in the GitHub Sponsorship Program, and another 34,000 people who were active GitHub users but did not enroll in the GitHub sponsorship program. Their email addresses were extracted from publicly available data from GitHub. This puts our set of users we intend to share our survey at 44,238 individuals.
However, not all of the 44,238 individuals are expected to respond. To estimate of our expected number of responses, we randomly sampled 1000 emails from our dataset and emailed our survey to these individuals. After a week, we had 17 responses in total. Then, we sent a final reminder, and increased our total number of responses to 36.
Since all our questions were optional, for our estimated response sizes, we focused on the respondents who answered the primary outcome questions. We see that 16-32 people answered our primary outcome questions. Based on this range, we created two estimates of our expected number of observations: [a] an optimistic estimate, and [b] a pessimistic estimate.
[a] Optimistic Estimate = (32/1000)*(44238-1000) = 1384 individuals
[b] Pessimistic Estimate = (16/1000)*(44238-1000) = 692 individuals
We plan to continue the approach of emailing individuals at 9 AM local time on a weekday, and then sending a final reminder email to those who did not respond a week after our initial email. Due to the large size of the set, we plan to randomly split the 43,238 users we plan to reach out to into 4 batches. We will then email our users in the following staggered approach:
[Thursday; First week] - Initial email sent out to first batch of 10,809 people
[Thursday; Second week] - Initial email sent out to second batch of 10,810 people; Reminder email sent to the first batch
[Thursday; Third week] - Initial email sent out to third batch of 10,809 people; Reminder email sent to the second batch
[Thursday; Fourth week] - Initial email sent out to fourth batch of 10,810 people; Reminder email sent to the third batch
[Thursday; Fifth week] - Reminder email sent to the fourth batch
[Thursday; Sixth week] - We use the responses collected as of this date for our analysis
Arms
Our intervention involves 4 treatment arms, as described in the registration above. Since each arm is randomly assigned to users, we expect an even split between the number of individuals assigned to each treatment. Based on the optimistic estimate described above of 1384 individuals, we expect 346 individuals to be assigned to each treatment group.
346 individuals are told to assume that they receive no sponsorship after 6 months of being sponsorable.
346 individuals are told to assume that they receive a $ 20 sponsorship after 6 months of being sponsorable.
346 individuals are told to assume that they receive a $ 1000 sponsorship after 6 months of being sponsorable.
346 individuals are told to assume that they receive sponsorship from an organization after 6 months of being sponsorable.
Primary outcomes
(Descriptive Question) The first question in our survey asks respondents to rate how important they consider different benefits of contributing to open source software in motivating them to contribute more. This is done on a Likert scale from 1 (Not Important At All) to 5 (Very Important). This information is collected for each of the following:
1. Advancement of technology and the benefit of society.
2. Recognition from other open-source developers.
3. Improving opportunities for future jobs and promotions.
4. Making an extra income.
(Intervention) Additionally, as described in our intervention above, we hope to understand the differences in motivation (based on a Likert scale from 1 [Least Motivated] to 5 [Most Motivated]) for each of the actions below by an individual based on what treatment they were provided:
1. Committing new code to existing open source repositories.
2. Resolving issues you find in others’ open source repositories.
3. Starting new original open source repositories from scratch.
4. Forking other open source repositories.
These outcome variables will be analyzed using linear regression models to assess the impact of different motivational factors on the respondents' likelihood of engaging in various open-source activities.
Secondary outcomes
We seek to understand the familiarity and opinions of GitHub users with the GitHub Sponsors Program through free-text responses, and other post-hoc survey questions.
We also report splits of our data by sponsorables vs non-sponsorables.
Additionally, we ask them what best describes how they contribute to open source, and provide them the following options to choose from:
1. I maintain the code in projects written by others, such as closing the issues created by others.
2. I am an opinion leader in my domain of expertise, and I actively contribute to discussions where critical decisions are made for some open-source projects.
3. I invest in documenting and solving issues in the code, including those that others experience.
4. I start new projects and identify new needs open source can address, and sometimes these projects may be carried by others later.
5. Other (Please Describe): __________
For those individuals that have received sponsorships under the GitHub sponsorship program, we ask them questions about how the sponsorship changed their contributions to open source. If an individual did not receive sponsorships under the program, we ask them if they know anyone who received sponsorships under the program, and how observing someone else receiving sponsorships changed their contributions to open source.
Through our intervention which varies the hypothetical amount and type of sponsorship received by the respondent, we aim to understand the effect of compensation on multiple labor outcomes (i.e. understand how their motivations change for different ways one can contribute to open source). This is evaluated on a Likert scale of 1 (Least Motivated) to 5 (Most Motivated).
The analyses will also compare differences between groups (e.g., sponsorables vs. non-sponsorables) to understand how motivation levels vary based on the respondent's background.
Cited by
No citation data available for this registration yet.