Courts of Tomorrow

How to cite

Mehmood, Sultan; Ash, Elliot. 2024. "Courts of Tomorrow." AEA RCT Registry. AEARCTR-0012906. https://doi.org/10.1257/rct.12906-1.2

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Sultan Mehmood* , Elliot Ash

* Primary investigator

Registry ID
AEARCTR-0012906
DOI
https://doi.org/10.1257/rct.12906-1.2
First registered
Last updated
Status
In development
Trial dates
2024-01-01 – 2025-12-31
Countries
Private
JEL codes
O33 Technological Change: Choices and Consequences • Diffusion Processes M53 Training K41 Litigation Process C45 Neural Networks and Related Topics
Registry record
http://www.socialscienceregistry.org/trials/12906

Abstract

We integrate a specialized AI course into the Pakistan Judicial Academy's "Technology and Law” initiative. We will design and build an AI-based judge support tool that empowers judges to search, cite, and summarize the history of Pakistan precedents as well as newly submitted briefs and other documents. The system, especially with the associated training, is protected against plagiarism, hallucination, and providing false citations. The tool and training will be provided in the context of a randomized field experiment, equipping about one-third of Pakistan's trial court judges with generative AI technology and associated training and support. Our research will evaluate the effect of AI technology and training on judge performance, including AI usage, perceived AI usefulness, and quality measures constructed from written rulings. Our study's findings have the possibility to shed light on the potential of generative AI to bolster state capabilities and judicial productivity worldwide.

*Update to Trial - Rerandomization in October 2024
The random assignment of judges was conducted in two distinct waves for registration of judges into JudgeGPT subscriptions.

In February 2024, the first wave of registration saw 979 judges from Pakistan's lower courts sign up to participate in our experiment. We randomly assigned these 979 judges into two groups: 487 judges were allocated to the treatment group (Batch 1) and provided with access to the JudgeGPT subscription and GPT instruction course, while the remaining 492 judges were designated as the control group (Batch 2), scheduled to receive the same access in September 2024. This setup allows for a randomized control trial comparing the outcomes of Batch 1 and Batch 2. Following the initial random assignment, the introduction of the password-protected JudgeGPT, designed specifically to prevent spillovers, sparked considerable interest among judges who had not initially registered for the course but were nonetheless eager to participate but could not access GPT or the course.

An additional 580 judges expressed interest in the course and the JudgeGPT tool. To preserve the study's integrity, we decided against adding these new applicants to our control group (Batch 2), as they were not randomly assigned. Therefore, a second randomization was conducted to maintain the integrity of the study and increase its statistical power, accommodating a total of 1559 judges instead of the initially registered 979. This means more than 50% of the trial court judges (court of first instance) in Pakistan registered to participate in our experiment.

In October, the second wave of randomization, therefore, took place on October 23, 2024, for 580 judges. The 580 judges were randomly assigned to Batch 3 (n = 218), which will take the course in December 2024 and January 2025, and Batch 4 (n = 362). Batch 3 judges would get the same treatment as Batch 1 and 2: JudgeGPT course and JudgeGPT subscription. Batch 4, however, is further randomized into two subgroups: Batch 4a and Batch 4b. Batch 4a is randomly assigned to receive JudgeGPT training and a placebo course on Technology and Law in December 2024 and January 2025, along with a GPT subscription (and an anti-hallucination warning in GPT). Batch 4b will also take the generic Technology and Law course during the same period but will not receive a GPT subscription. The key difference is that Batch 4a will have access to the GPT subscription with a hallucination warning, while Batch 4b will not. Both groups, however, will attend the Generic Law and Technology classes at the same time that Batch 3 is receiving the JudgeGPT course. This will allow us to assess the impact of access to GPT tools on judges' learning and decision-making. Please see Figure 1 and other details in the Pre-analysis plan document for a summary of the experimental design and more details.

Intervention

We recruited about one-third of trial court judges in Pakistan to participate in a judge training course where they will receive access to a judge support chatbot and get training in using this tool. The study will employ a randomized controlled trial design, where about half of the judges will be randomized into a first round of the course, and the other half will be randomized into a second round about 8 months later.

Experimental design

The study will employ a randomized controlled trial design, where about half of the judges will be randomized into a first round of the course, and the other half will be randomized into a second round in the next course term.

Design details

Randomization method
Randomization is done by a computer.
Randomization unit
We recruited about one-third of trial court judges in Pakistan (N=800) to participate. The study will employ a randomized controlled trial design, where about half of the judges (N=400) will be randomized into the first round of the course in February-March 2024, and the other half (N=400) will be randomized into the second round of the course in September--October 2024. Stratification is based on the time of enrollment in the course, the age of judges, and the district court in which the judges serve.
Clusters
Standard errors will be clustered by stratification units.
Observations
760 judges
Arms
760 judges
Primary outcomes
1. Course attendance and participation. 2. Use of the AI tool. 3. Judgment Quality using Judgments Texts Data. The Federal Judicial Academy will share judgment texts on each judge that will allow us to construct the following main text-based measures:. 4. Perceptions of AI Utility 5. Productivity. Judges’ expectations of AI's potential to improve their productivity and their support for using generative AI at work.
Secondary outcomes
1. Case Backlog and Disposal Rate. This administrative data will be procured from the Judicial Academy and the Pakistan Law and Justice Commission of Pakistan if and when it becomes available. 2. Workload Management. Measuring the number of different cases a judge works on at any given time, along with the hours spent on work.

Cited by

No citation data available for this registration yet.

Keywords