The Impact of Specialized AI tools for Lawyering Tasks

How to cite

Manning, Sam; Schwarz, Daniel; Prescott, JJ; Cleveland, David; Barry, Patrick; Rich, Beverly. 2024. "The Impact of Specialized AI tools for Lawyering Tasks." AEA RCT Registry. AEARCTR-0014957. https://doi.org/10.1257/rct.14957-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Sam Manning* , Daniel Schwarz , JJ Prescott , David Cleveland , Patrick Barry , Beverly Rich

* Primary investigator

Registry ID
AEARCTR-0014957
DOI
https://doi.org/10.1257/rct.14957-1.0
First registered
Last updated
Status
Completed
Trial dates
2024-10-01 – 2024-11-04
Countries
United States of America (Minnesota and Michigan)
Registry record
http://www.socialscienceregistry.org/trials/14957

Abstract

We seek to study the impact of large language models (LLMs) and LLMs with additional domain-specific scaffolding on legal work. While pre-trained LLMs have been shown to increase worker productivity across a number of tasks, there is limited empirical research on how domain-specific software integrations and post-training model enhancements could drive additional impacts on worker productivity and quality. In real-world work applications, LLMs are often enhanced with “scaffolding” that includes tool-use capabilities, retrieval-augmented generation (RAG), access to specialized datasets, backend prompt optimization, and user interfaces tailored for specific use cases. Recent research suggests that such scaffolding can significantly enhance the capabilities of pre-trained models, but there is a lack of empirical studies quantifying the effect this has on different kinds of work. As AI progress continues, understanding the marginal impact of scaffolding built around LLMs will be essential to gauging AI’s impact on the future of work. This study aims to generate new evidence on this question, using legal work as an experimental setting. To do so, we propose a RCT to evaluate the impact of LLM-powered legal assistant software on legal labor. We will randomly assign second- and third-year law students into a no-AI control group and two treatment groups: one with access to OpenAI’s o1 model and the other with access to VLex’s Vincent AI, a legal assistant software built on GPT-4o with additional features specific to legal work. We will then measure productivity and work quality differences on a set of 6 legal work tasks that participants complete, and estimate the causal impact of using LLMs and LLM-powered tools to assist with these tasks.

Intervention

Participants will be asked to complete 6 different legal research tasks that are illustrative of the type of work that a junior associate normally needs to perform on the job. Access to an AI assistant tool (either OpenAI's o1-preview model or Vincent AI's legal research tool or no AI tool) will be randomly assigned for the task. The trial aims to estimate the causal impact of access to each AI tool on worker productivity and quality on legal tasks.

Experimental design

Participants consent to enroll in the study and then are randomly assigned into one of three groups: A, B, or C. Each group has different stipulations around access for Vincent AI, OpenAI's o1-preview model, or no AI to assist them while completing the assigned legal work task.

Group A:

Assignment 1: No AI
Assignment 2: Vincent AI
Assignment 3: o1
Assignment 4: No AI
Assignment 5: Vincent AI
Assignment 6: o1

Group B:

Assignment 1: o1
Assignment 2: No AI
Assignment 3: Vincent AI
Assignment 4: o1
Assignment 5: No AI
Assignment 6: Vincent AI

Group C:

Assignment 1: Vincent AI
Assignment 2: o1
Assignment 3: No AI
Assignment 4: Vincent AI
Assignment 5: o1
Assignment 6: No AI

Design details

Randomization method
Randomization done by a computer. Randomly sorted observations and split observations into thirds to assign group numbers to participants.
Randomization unit
Individual
Clusters
1, this was not a clustered randomization.
Observations
153
Arms
51 in Group A, 51 in Group B, 51 in Group C. This means that 51 participants completed each of the six tasks with no AI, access to o1-preview, and Vincent AI, respectively.
Primary outcomes
Work Quality: For each of the six tasks, we will evaluate treatment effects on the following criteria that make up work quality: Accuracy Analysis Organization Clarity Professionalism Each will be graded on a 1-7 scale by law professors who are given detailed grading rubrics and instructions. We will assess a measure overall quality that is an average of the scores on each component of quality. Time: A self-reported measure of the number of minutes it took to complete each task. Productivity: Overall quality divided by number of minutes it took to complete the task.
Secondary outcomes
We may also perform subgroup analysis outcome measures across groups of similar tasks (as opposed to an average quality rating for all tasks). We may also perform heterogeneity analysis by participant gender, university, or other demographic variables. We may also compare results to results in Choi et al 2023 that examined the impact of GPT-4 on legal work tasks, but this analysis would be exploratory and interpretation will need to account for the fact that the Choi et al 2023 used a different set of legal work tasks.

Cited by

No citation data available for this registration yet.

Keywords