Assessing Generative AI value in a public sector context: evidence from a field experiment

How to cite

Fitzpatrick, Trevor; Walsh, David; Kelly, Seamus; Carey, Patrick; Nugent, Ruairi; Cleary, Tom. 2024. "Assessing Generative AI value in a public sector context: evidence from a field experiment." AEA RCT Registry. AEARCTR-0013347. https://doi.org/10.1257/rct.13347-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Trevor Fitzpatrick* , David Walsh , Seamus Kelly , Patrick Carey , Ruairi Nugent , Tom Cleary

* Primary investigator

Registry ID
AEARCTR-0013347
DOI
https://doi.org/10.1257/rct.13347-1.0
First registered
Last updated
Status
In development
Trial dates
2024-04-24 – 2024-07-10
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/13347

Abstract

Generative AI economic benefits in aggregate are often cited. However, the productivity or efficiency improvements within organisations is an emerging topic. Recent research focused on knowledge work suggest productivity or efficiency improvements may be large, finding significant effects for certain types of tasks, with more benefits to less experienced workers. Evaluating the value/benefits and costs of GenAI in knowledge-driven organisations within a public sector may not be as clear-cut in a for-profit context. To provide some evidence, our study will test the impact of GenAI on two knowledge-work tasks among staff in the organisation.

Intervention

The potential productivity or efficiency improvements of Generative AI within organisations is an emerging research topic, including the potential effects for knowledge workers. Recent research (Dell'Acqua et al., 2023) focused on knowledge work suggest productivity or efficiency improvements may be large, finding significant effects for certain types of tasks, with more benefits to less experienced staff. To provide initial evidence, our study will test the impact of GenAI on two representative knowledge-work tasks among staff in the organisation. The first task is data related, and the second task is document based. These differ from the types of tasks considered so far in the literature.

Experimental design

A two phase design, with a pre-assessment task to baseline individual subjects and two tasks within the main experiment. The control group receives task instructions, data or documents to work from. The treatment group receives task instructions, data/documents to work from and the assistance of an LLM focused on these types of tasks.

Design details

Randomization method
Done in office by computer
Randomization unit
Subject
Clusters
0
Observations
152
Arms
38 for treatment (LLM use); 38 control (no LLM use) for each task
Primary outcomes
The key outcome variable for each participant is quality of the response measured by the overall quality rating (graded by independent grader's according to a rubric) and objective measures (% of tasks completed correctly)
Secondary outcomes
Task Time (duration in mins/seconds)

Cited by

No citation data available for this registration yet.

Keywords