Domain-specific knowledge and LLM performance

How to cite

Streich, David; Töllich, Niklas; Hornuf, Lars. 2024. "Domain-specific knowledge and LLM performance." AEA RCT Registry. AEARCTR-0014755. https://doi.org/10.1257/rct.14755-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

David Streich* , Niklas Töllich , Lars Hornuf

* Primary investigator

Registry ID
AEARCTR-0014755
DOI
https://doi.org/10.1257/rct.14755-1.0
First registered
Last updated
Status
In development
Trial dates
2024-11-02 – 2024-11-30
Countries
Private
JEL codes
G11, G14, G50
Registry record
http://www.socialscienceregistry.org/trials/14755

Abstract

Large language models (LLMs), the most well-known form of generative artificial intelligence, have proven remarkably useful in the Finance domain. Thus far, studies on the capabilities of LLMs in financial applications have investigated general-purpose models (mostly drawing on OpenAI's cutting-edge GPT models) without domain-specific knowledge (i.e., Finance-specific knowledge). It stands to reason that the performance of LLMs could be further enhanced by adding domain-specific knowledge to pre-trained LLMs. Thus, we study how injecting domain-specific knowledge affects the performance of LLMs in a simple portfolio allocation task. We vary the type and scope of domain-specific information we inject and compare the performance across treatment conditions and to a human benchmark derived from an incentivized survey among financial advisors.

Intervention

We will vary the type and scope of domain-specific information injected to the LLMs. More information is provided in the hidden intervention description.

Experimental design

Portfolio recommendations will be elicited from 7 state-of-the-art LLMs, for 12 hypothetical investor profiles and in 5 experimental conditions. As a human benchmark, we elicit portfolio recommendations for each of the 12 profiles from 100 human financial advisors in an incentivized online survey. More information is provided in the hidden experimental design description.

Design details

Randomization method
Randomization done in office by a computer
Randomization unit
Not applicable
Clusters
7 LLMs + 1 platform for human advisor recommendations (100 advisors targeted)
Observations
1620 (7 LLMs * 12 profiles * 5 experimental conditions = 420 LLM-generated observations; 100 financial advisors * 12 profiles = 1200 human-generated observations)
Arms
84 observations (7 LLMs * 12 profiles) per experimental condition in the LLM arm; 1200 observations (100 advisors * 12 profiles) in the human financial advisor arm.
Primary outcomes
Stock-level analyses: individual portfolio weights, portfolio weights of groups of stocks (e.g., value stocks, large stocks, etc.) Portfolio-level analyses: diversification measures (e.g., HHI, number of securities, risky share, IVOL), risk-adjusted performance (e.g., excess returns, Sharpe ratio, CAPM alpha, FF6 alpha) both in and out of sample.

Cited by

No citation data available for this registration yet.

Keywords