The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers

How to cite

peng, sida. 2024. "The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers." AEA RCT Registry. AEARCTR-0014530. https://doi.org/10.1257/rct.14530-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

sida peng*

* Primary investigator

Registry ID
AEARCTR-0014530
DOI
https://doi.org/10.1257/rct.14530-1.0
First registered
Last updated
Status
Completed
Trial dates
2022-01-01 – 2024-05-05
Countries
United States of America ()
JEL codes
J24
Registry record
http://www.socialscienceregistry.org/trials/14530

Abstract

This study evaluates the impact of generative AI on software developer productivity by
analyzing data from three randomized controlled trials conducted at Microsoft, Accenture,
and an anonymous Fortune 100 electronics manufacturing company. These field
experiments, which were run by the companies as part of their ordinary course of business,
provided a randomly selected subset of developers with access to GitHub Copilot, an AIbased
coding assistant that suggests intelligent code completions.

Intervention

The study granted a randomly selected subset of software developers access to GitHub
Copilot, an AI-based coding assistant that suggests intelligent code completions.

Experimental design

Initially, a random subset of developers is granted access to GitHub Copilot. During the
intervention period, developers not in our treatment group cannot use Copilot. After the
intervention period, all developers are able to use Copilot, but there may still be differential
adoption across treatment and control groups, and we also utilize this variation.

Design details

Randomization method
computer-based randomization
Randomization unit
For 50% of teams at Microsoft, the treatment is randomized by team; for the remaining 50%, it is assigned randomized by individual. At Accenture, the treatment is randomized individually. At anonymous company, the treatment is clustered by team.
Clusters
1438
Observations
4869
Arms
Microsoft: 70 teams assigned to treat, 79 assigned to control; for remaining 152 teams, randomization is individual with 755 individuals treated, 766 individuals control. Accenture: 193 developers treated, 123 developers in control Anon. Company: roughly equally split 432 teams into 6 rollout weeks (each week, another ~16% of teams get access)
Primary outcomes
# Pull Requests, # Commits, # Builds, Build Success Rate, # Accepted Suggestions, Fraction of Developers Using Copilot (i.e., Adoption)
Sponsors
Name: Microsoft Location: Boston Url: https://microsoft.com
Partners
Name: Microsoft Partner_type: private_company Url: https://microsoft.com

Cited by

No citation data available for this registration yet.

Keywords