A Field Experiment on the Future of Software Development
How to cite
Vendraminelli, Luca; Sawarni, Ayush; Bissett, Patrick; Padmakumar, Vishakh; Haupt, Andreas; Bennett, Logan; Jin, Wang; Karunakaran, Arvind; Yang, Diyi; Syrgkanis, Vasilis; Poldrack, Russell; Brynjolfsson, Erik. 2026. "A Field Experiment on the Future of Software Development." AEA RCT Registry. AEARCTR-0019333. https://doi.org/10.1257/rct.19333-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Artificial intelligence may lower the technical barrier to writing code and change which skills make a software developer effective. We study this question in a one-day field experiment with employees of a major IT services and consulting firm based in India. Participants work for two 3.5-hour blocks, each containing seven programming tasks that implement features in an internal application whose source code was never published online, and their code is graded by unit tests they cannot see. We randomise two treatments. First, each participant is assigned to one of three AI-access arms: No AI, Weaker AI (GitHub Copilot restricted to Claude Haiku), or Stronger AI (GitHub Copilot with Claude Haiku, Sonnet, and Opus). Second, within participants, one block presents the task requirements as formal assignments and the other embeds the same requirements in transcripts of interviews with a non-technical stakeholder, with the order randomised. The randomised population is the 6,806 registered employees who completed at least one pre-experimental assessment. Assignment to AI access uses permuted blocks within role-type by tenure strata, with arm probabilities that differ by role type, and we expect approximately 5,000 participants to attend. Before the experiment day, every participant completes a battery that measures six components of expertise: coding knowledge, career experience, on-the-job coding time, AI expertise, fluid intelligence, and working memory. The primary outcome is the block score, the share of rubric points earned across the block's tasks. The confirmatory hypotheses test whether AI access raises performance, whether its effect is smaller when requirements arrive as an interview transcript, and whether it grows with each expertise component, so that AI shifts the relative value of accumulated knowledge and cognitive traits. A baseline in which the same models attempt every task without human involvement benchmarks what each AI tier achieves alone.
Intervention
The experiment runs on a single day at the partner firm's sites in India. Participants work on two blocks of seven programming tasks each, with 3.5 hours per block and a 1.5-hour break between the blocks. The tasks implement features in an internally developed document-management application whose source code has never been published online, so the tasks are novel both for the participants and for the AI models. Participants write code in Visual Studio Code and submit on an experimental platform as often as they wish within the block. After each submission the platform runs the task's unit tests on the submitted code and returns only the number of tests passed. The test code itself is never shown.
The first intervention is AI access, randomised between participants in three arms. In the No AI arm, participants work without any AI assistance. In the Weaker AI arm, participants receive GitHub Copilot restricted to Claude Haiku. In the Stronger AI arm, participants receive GitHub Copilot with Claude Haiku, Claude Sonnet, and Claude Opus. Participants in the two AI arms receive a 100-dollar AI credit budget per block, refreshed during the break.
The second intervention is the format of the task requirements, varied within participants across the two blocks. One block presents each task as a formal assignment, a structured specification that states the goal, the scope, and every rule and parameter explicitly. The other block presents each task as the transcript of an interview in which a non-technical product stakeholder describes what they need in informal language. Both versions carry the information needed to pass every unit test, so the formats differ in how the requirements are presented and the interview format requires the participant to extract and formalise them before coding. Half the participants receive the formal block first and half receive the interview block first.
Experimental design
Population and stratification.
The partner firm registered its employees for the study through a recruitment grid crossing role type with tenure. Role types follow the firm's records: Experts (employees on a leadership and technical-authority track), Developers (employees hired to deliver code-based assignments), and Non-coders (employees in adjacent functions such as sales and marketing, typically without formal coding skills). Tenure has three levels: junior (under 3 years), mid (3 to 8 years), and senior (8 years or more). The randomised population is the 6,806 registrants who completed at least one pre-experimental assessment.
Treatments.
Two treatments are randomised independently. AI access is a between-participant factor with three arms (No AI, Weaker AI, Stronger AI, described in the Intervention field). Requirement format is a within-participant crossover: each participant completes one block with formal assignments and one block with stakeholder-interview transcripts, and the order is randomised with probability one half. Experts and Developers are assigned to No AI with probability 1/5 and to each AI arm with probability 2/5. Non-coders are assigned only to the two AI arms with probability 1/2 each, because asking participants without coding skills to complete programming tasks without AI would be unfair and demotivating; their No AI performance is set to zero. All participants receive the same two blocks in the same order (the block content is fixed, the format is not), the seven tasks within a block appear in a randomly shuffled order per participant, and participants choose freely which task to attempt and when.
Tasks.
The 14 experimental tasks were selected from 20 candidates developed with the firm's engineering team, each with unit tests and a scoring rubric validated by the firm. Tasks that the strongest available model solved at the first attempt from the pasted formal assignment were removed, and the remaining tasks were split into two blocks balanced on a machine difficulty index (from an LLM-alone baseline) and a human difficulty index (from expert rankings).
Pre-experimental measurement.
Before the experiment day, every participant completes a battery of assessments: a registration survey, a professional-experience survey, a coding-skills survey, a timed 15-item coding assessment, an LLM-expertise survey, an LLM-driven career interview, and a cognitive and personality battery (Raven's Progressive Matrices, Digit Span, Cognitive Flexibility Inventory, Ten-Item Personality Inventory, General Risk Propensity Scale, and a delay-discounting task). Six confirmatory expertise measures come from this battery: coding knowledge (the timed coding assessment score), career experience (total professional experience in years), on-the-job coding time (the share of the working week spent on the work the tasks exercise), AI expertise (a five-component index of GenAI knowledge and use), fluid intelligence (Raven's score), and working memory (total Digit Span).
Hypotheses.
We test nine confirmatory hypotheses.
- H1: the effect of AI on task performance is positive, and larger for Stronger than for Weaker AI.
- H2: the effect of AI is smaller when requirements arrive as a stakeholder interview rather than a formal assignment, and more so for Stronger AI.
- H3 to H8: the effect of AI is greater when coding knowledge (H3), career experience (H4), on-the-job coding time (H5), AI expertise (H6), fluid intelligence (H7), or working memory (H8) is higher, in each case more so for Stronger AI.
- H9: AI access transforms the relative value of knowledge and cognitive traits, so the effect of AI grows more with an index of cognitive traits (fluid intelligence and working memory) than with an index of knowledge (coding knowledge, on-the-job coding time, and career experience).
Estimation.
The hypotheses are tested in one ordinary-least-squares regression per block. The block score is regressed on the two AI-arm indicators, the format indicator, the six z-scored expertise measures, and the full set of their interactions. H9 uses a companion regression per block that replaces the six measures with three range-normalised indexes (cognitive traits, knowledge, AI expertise). An LLM-alone baseline, in which the same models attempt every task under both formats with three prompt scripts and three repetitions, provides each task's machine difficulty index and the benchmark for value added.
Exploratory analysis.
Heterogeneous effects of both treatments are estimated with a post-selection Lasso (with selective inference) and an honest causal forest over the pre-treatment covariates, with best-linear-projection, group-average-effect, and RATE summaries. Primary analyses include every participant who was assigned a condition; pre-specified robustness subsets are participants with at least one valid submission, participants meeting the engagement threshold, and each participant's first block only.
Design details
Cited by
No citation data available for this registration yet.