Domain-specific knowledge and LLM performance
How to cite
Streich, David; Töllich, Niklas; Hornuf, Lars. 2024. "Domain-specific knowledge and LLM performance." AEA RCT Registry. AEARCTR-0014755. https://doi.org/10.1257/rct.14755-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Large language models (LLMs), the most well-known form of generative artificial intelligence, have proven remarkably useful in the Finance domain. Thus far, studies on the capabilities of LLMs in financial applications have investigated general-purpose models (mostly drawing on OpenAI's cutting-edge GPT models) without domain-specific knowledge (i.e., Finance-specific knowledge). It stands to reason that the performance of LLMs could be further enhanced by adding domain-specific knowledge to pre-trained LLMs. Thus, we study how injecting domain-specific knowledge affects the performance of LLMs in a simple portfolio allocation task. We vary the type and scope of domain-specific information we inject and compare the performance across treatment conditions and to a human benchmark derived from an incentivized survey among financial advisors.
Intervention
We will vary the type and scope of domain-specific information injected to the LLMs. More information is provided in the hidden intervention description.
Experimental design
Portfolio recommendations will be elicited from 7 state-of-the-art LLMs, for 12 hypothetical investor profiles and in 5 experimental conditions. As a human benchmark, we elicit portfolio recommendations for each of the 12 profiles from 100 human financial advisors in an incentivized online survey. More information is provided in the hidden experimental design description.
Design details
Cited by
No citation data available for this registration yet.