Experimental Evidence on the Productivity Effects of Large Language Models

How to cite

Noy, Shakked; Zhang, Whitney. 2023. "Experimental Evidence on the Productivity Effects of Large Language Models." AEA RCT Registry. AEARCTR-0010882. https://doi.org/10.1257/rct.10882-2.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Shakked Noy* , Whitney Zhang

* Primary investigator

Registry ID
AEARCTR-0010882
DOI
https://doi.org/10.1257/rct.10882-2.1
First registered
Last updated
Status
Completed
Trial dates
2023-01-27 – 2023-02-24
Countries
United States of America ()
Registry record
http://www.socialscienceregistry.org/trials/10882

Abstract

We examine the productivity effects of a new artificial intelligence technology—large language models—in the context of creative tasks. In an online experiment, we assign occupation-specific writing tasks to a variety of college-educated professionals, and randomly expose half of them to the assistive chatbot ChatGPT. We examine takeup of ChatGPT, effects on productivity and output quality, complementarity with baseline ability, effects on task structure, skill demand, and the distribution of ideas, and effects on job satisfaction, self-efficacy, and automation concerns.

Intervention

600 professionals on Prolific complete two occupation-specific incentivized writing tasks. Between the first and second task, the treatment group is asked to sign up for a ChatGPT account.

Experimental design

Details of experimental design will become public when the trial finishes.

Design details

Randomization method
Qualtrics
Randomization unit
Individual respondent
Clusters
600
Observations
600 respondents, 1200 task completions
Arms
50/50 treatment/control orthogonal cross-randomization: 40/40/20 linear/convex/exact time incentives
Primary outcomes
See uploaded analysis plan for full specification of regressions. Outcomes: - Grades - Time spent - Earnings per minute, incl. normalized by Prolific workers' outside option - Subjective usefulness of ChatGPT, willingness to pay for ChatGPT in job
Secondary outcomes
Job satisfaction, self-efficacy, concerns about automation, followup survey results

Cited by

No citation data available for this registration yet.

Keywords