LLMs in Strategic Decision Making: Problem Formulation and Solution Ideation

How to cite

Kim, Hyunjin; Wu, Nety; Lin, Chengyi. 2024. "LLMs in Strategic Decision Making: Problem Formulation and Solution Ideation." AEA RCT Registry. AEARCTR-0013044. https://doi.org/10.1257/rct.13044-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Hyunjin Kim* , Nety Wu , Chengyi Lin

* Primary investigator

Registry ID
AEARCTR-0013044
DOI
https://doi.org/10.1257/rct.13044-1.0
First registered
Last updated
Status
In development
Trial dates
2024-02-15 – 2025-02-15
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/13044

Abstract

This project aims to understand the influence of employing a Large Language Model (LLM) on problem formulation and solution ideation stages during the strategic decision making process.

Experimental design

This project aims to understand the influence of employing a Large Language Model (LLM) on problem formulation and solution ideation stages during the strategic decision making process.

Design details

Randomization method
Individuals are randomized into control and treatment by stratifying on gender and industry (whether they worked in consulting), using a computer.
Randomization unit
Randomization is conducted at the individual level. Each participant is first randomly assigned to one of the three conditions, stratified on gender and industry (whether they worked in consulting).
Clusters
N/A
Observations
336 students
Arms
112 participants control, 112 participants treatment 1, 112 participants treatment 2.
Primary outcomes
All participants will be asked to articulate a problem statement and their thought process in developing the problem statement. We will analyze the length, depth, and breadth of this problem statement, and how they prioritize the problems, if they identify multiple. We will also categorize their response on the thought process using NLP techniques. From the options they input, we will observe the following outcomes: (1) the number of total options that the student brainstormed (2) the number/share of strategic (rather than operational) options that the student brainstormed (3) a binary measure of whether the options were mutually exclusive (4) the number/share of options that suggest that the company continue its current strategy (vs. exit or expand) (5) binary measures of whether the best-chosen option is to "continue", "exit", or "expand" In addition, we will measure the time they used to complete the task, and the cognitive load of the participant using a math question at the end of the experiment.
Secondary outcomes
Survey questions to all participants: (1) Describe how you developed your options; (2) How confident are you in the options you have developed? (3) How would you describe the difficulty of the task? (4) Frequency of using LLMs and familiarity with LLMs. Additional survey questions to the treatment group only: (1) Did you find the LLM tool provided useful? (2) Please elaborate on why it was useful or not useful. Peer evaluations of option quality, binary variables on how detailed the option is, various details about the nature of the option, and group-level outcomes on strategic options. Secondary outcomes will be collected and coded based on feasibility (e.g., if budget allows).

Cited by

No citation data available for this registration yet.

Keywords