The Impact of Specialized AI tools for Lawyering Tasks
How to cite
Manning, Sam; Schwarz, Daniel; Prescott, JJ; Cleveland, David; Barry, Patrick; Rich, Beverly. 2024. "The Impact of Specialized AI tools for Lawyering Tasks." AEA RCT Registry. AEARCTR-0014957. https://doi.org/10.1257/rct.14957-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
We seek to study the impact of large language models (LLMs) and LLMs with additional domain-specific scaffolding on legal work. While pre-trained LLMs have been shown to increase worker productivity across a number of tasks, there is limited empirical research on how domain-specific software integrations and post-training model enhancements could drive additional impacts on worker productivity and quality. In real-world work applications, LLMs are often enhanced with “scaffolding” that includes tool-use capabilities, retrieval-augmented generation (RAG), access to specialized datasets, backend prompt optimization, and user interfaces tailored for specific use cases. Recent research suggests that such scaffolding can significantly enhance the capabilities of pre-trained models, but there is a lack of empirical studies quantifying the effect this has on different kinds of work. As AI progress continues, understanding the marginal impact of scaffolding built around LLMs will be essential to gauging AI’s impact on the future of work. This study aims to generate new evidence on this question, using legal work as an experimental setting. To do so, we propose a RCT to evaluate the impact of LLM-powered legal assistant software on legal labor. We will randomly assign second- and third-year law students into a no-AI control group and two treatment groups: one with access to OpenAI’s o1 model and the other with access to VLex’s Vincent AI, a legal assistant software built on GPT-4o with additional features specific to legal work. We will then measure productivity and work quality differences on a set of 6 legal work tasks that participants complete, and estimate the causal impact of using LLMs and LLM-powered tools to assist with these tasks.
Intervention
Participants will be asked to complete 6 different legal research tasks that are illustrative of the type of work that a junior associate normally needs to perform on the job. Access to an AI assistant tool (either OpenAI's o1-preview model or Vincent AI's legal research tool or no AI tool) will be randomly assigned for the task. The trial aims to estimate the causal impact of access to each AI tool on worker productivity and quality on legal tasks.
Experimental design
Participants consent to enroll in the study and then are randomly assigned into one of three groups: A, B, or C. Each group has different stipulations around access for Vincent AI, OpenAI's o1-preview model, or no AI to assist them while completing the assigned legal work task.
Group A:
Assignment 1: No AI
Assignment 2: Vincent AI
Assignment 3: o1
Assignment 4: No AI
Assignment 5: Vincent AI
Assignment 6: o1
Group B:
Assignment 1: o1
Assignment 2: No AI
Assignment 3: Vincent AI
Assignment 4: o1
Assignment 5: No AI
Assignment 6: Vincent AI
Group C:
Assignment 1: Vincent AI
Assignment 2: o1
Assignment 3: No AI
Assignment 4: Vincent AI
Assignment 5: o1
Assignment 6: No AI
Design details
Cited by
No citation data available for this registration yet.