This project aims to understand the influence of employing a Large Language Model (LLM) on problem formulation and solution ideation stages during the strategic decision making process.
Experimental design
This project aims to understand the influence of employing a Large Language Model (LLM) on problem formulation and solution ideation stages during the strategic decision making process.
Design details
Randomization method
Individuals are randomized into control and treatment by stratifying on gender and industry (whether they worked in consulting), using a computer.
Randomization unit
Randomization is conducted at the individual level. Each participant is first randomly assigned to one of the three conditions, stratified on gender and industry (whether they worked in consulting).
All participants will be asked to articulate a problem statement and their thought process in developing the problem statement. We will analyze the length, depth, and breadth of this problem statement, and how they prioritize the problems, if they identify multiple. We will also categorize their response on the thought process using NLP techniques.
From the options they input, we will observe the following outcomes:
(1) the number of total options that the student brainstormed
(2) the number/share of strategic (rather than operational) options that the student brainstormed
(3) a binary measure of whether the options were mutually exclusive
(4) the number/share of options that suggest that the company continue its current strategy (vs. exit or expand)
(5) binary measures of whether the best-chosen option is to "continue", "exit", or "expand"
In addition, we will measure the time they used to complete the task, and the cognitive load of the participant using a math question at the end of the experiment.
Secondary outcomes
Survey questions to all participants: (1) Describe how you developed your options; (2) How confident are you in the options you have developed? (3) How would you describe the difficulty of the task? (4) Frequency of using LLMs and familiarity with LLMs.
Additional survey questions to the treatment group only: (1) Did you find the LLM tool provided useful? (2) Please elaborate on why it was useful or not useful.
Peer evaluations of option quality, binary variables on how detailed the option is, various details about the nature of the option, and group-level outcomes on strategic options.
Secondary outcomes will be collected and coded based on feasibility (e.g., if budget allows).
Cited by
No citation data available for this registration yet.