A Design Centric View of AI Enhanced Decision Making

How to cite

Kazemi, Saeid; Camuffo, Arnaldo; Gambardella, Alfonso; Pandey, Abhinav. 2025. "A Design Centric View of AI Enhanced Decision Making." AEA RCT Registry. AEARCTR-0014579. https://doi.org/10.1257/rct.14579-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Saeid Kazemi* , Arnaldo Camuffo , Alfonso Gambardella , Abhinav Pandey

* Primary investigator

Registry ID
AEARCTR-0014579
DOI
https://doi.org/10.1257/rct.14579-1.0
First registered
Last updated
Status
Completed
Trial dates
2025-04-21 – 2025-04-22
Countries
Italy (Lombardia)
Registry record
http://www.socialscienceregistry.org/trials/14579

Abstract

This study investigates how different forms of artificial intelligence (AI) support influence the quality of strategic decision-making under uncertainty. Using a 3×2×2 factorial design, we randomly assign participants to one of twelve experimental arms varying across (i) the type of AI assistance (none, general-purpose AI, or specialized agentic AI trained on the theory-based view), (ii) the plausibility of the decision problem (plausible vs. implausible), and (iii) the number of hypotheses participants are allowed to generate (single vs. multiple). Participants are asked to develop theories to solve a strategic problem, and we measure their subjective probability of success, the quality and novelty of their theories, and their confidence in those theories.
This design allows us to isolate both the direct effects and interaction effects of AI assistance, problem complexity, and hypothesis generation, offering insights into how organizations can best leverage AI tools to enhance strategic reasoning and innovation in uncertain environments.

Intervention

Interventions

In this study, participants will be randomly assigned to one of 12 experimental groups (blocked minimized randomization, due to the different levels of difficulty and therefore attrition rate across arms) to examine the effects of different types of AI assistance on decision-making and the development of theories.

Experimental Arms (12 arms)

Plausible Challenge (An easier to solve problem)

1. Control – Single Hypothesis (Group 1a): No Additional Assistance

Intervention: Participants will receive a plausible decision problem and have a single text box to enter their theorizing without any additional assistance.

Objective: Baseline measure of theorizing with a constrained single hypothesis.

2. Control – Multiple Hypotheses (Group 1b): No Additional Assistance

Intervention: Participants will receive a plausible decision problem and can add multiple text boxes to develop several theories without additional assistance.

Objective: Measure the impact of generating multiple hypotheses on theorizing without external assistance.

3. General AI Assistance – Single Hypothesis (Group 2a): Access to ChatGPT

Intervention: Participants will use ChatGPT O3-mini for assistance in brainstorming and refining their theorizing in response to a plausible problem, limited to one hypothesis.

Objective: Evaluate general AI impact on theorizing when constrained to a single hypothesis.

4. General AI Assistance – Multiple Hypotheses (Group 2b): Access to ChatGPT

Intervention: Participants will use ChatGPT O3-mini to develop and refine multiple hypotheses, having multiple text boxes available for entry.

Objective: Assess how general AI assistance combined with multiple hypotheses affects theory.

5. Specialized Agentic AI Assistance – Single Hypothesis (Group 3a): Access to agentic AI (TBV-trained AI)

Intervention: Participants will have access to agentic AI to generate and refine a single theory aligned with the theory-based view.

Objective: Investigate the specialized AI's impact on theorizing when limited to one hypothesis.

6. Specialized Agentic AI Assistance – Multiple Hypotheses (Group 3b): Access to agentic AI (TBV-trained AI)

Intervention: Participants will use agentic AI to generate, refine, and potentially autogenerate multiple theories.

Objective: Evaluate the effectiveness of specialized AI support in enhancing theorizing through multiple hypothesis generation.

Implausible Challenge (A more difficult to address problem)

7. Control – Single Hypothesis (Group 4a): No Additional Assistance

Intervention: Participants will receive an implausible decision problem and enter their theory using a single text box without assistance.

Objective: Baseline measure for implausible problems with a single hypothesis constraint.

8. Control – Multiple Hypotheses (Group 4b): No Additional Assistance

Intervention: Participants will tackle an implausible problem, generating multiple hypotheses in multiple text boxes without external assistance.

Objective: Examine how multiple hypotheses generation without assistance influences theorizing in challenging scenarios.

9. General AI Assistance – Single Hypothesis (Group 5a): Access to ChatGPT

Intervention: Participants facing an implausible problem will use ChatGPT O3-mini to assist in refining a single hypothesis.

Objective: Assess the impact of general AI assistance in challenging scenarios when constrained to one hypothesis.

10. General AI Assistance – Multiple Hypotheses (Group 5b): Access to ChatGPT

Intervention: Participants will leverage ChatGPT O3-mini to brainstorm, refine, and develop multiple theories for an implausible problem.

Objective: Evaluate the advantage of general AI support combined with multiple hypothesis generation in difficult problem settings.

11. Specialized Agentic AI Assistance – Single Hypothesis (Group 6a): Access to agentic AI (TBV-trained AI)

Intervention: Participants will use agentic AI to generate and refine one theory in response to an implausible problem.

Objective: Determine specialized AI's effectiveness in enhancing theorizing under single-hypothesis constraints in difficult scenarios.

12. Specialized Agentic AI Assistance – Multiple Hypotheses (Group 6b): Access to agentic AI (TBV-trained AI)

Intervention: Participants will utilize agentic AI to generate and refine multiple hypotheses, including autogenerated theories, in response to an implausible problem.

Objective: Investigate how specialized AI assistance maximizes theory development effectiveness through multiple hypothesis generation in complex contexts.

Purpose of Interventions

The interventions are designed to assess the impact of different types of AI assistance on prior formation, specifically focusing on:

The number of hypotheses n

The quality of the theory z

The confidence in the theory ω

Expected Success Probability π

And interacting these effects with the plausibility or implausibility of the problem.

By comparing these outcomes across the three groups, the study aims to provide insights into how various forms of AI assistance can influence theories and decision-making. This understanding is valuable for organizations considering the integration of AI support tools to enhance decision-making and innovation

Experimental design

This study utilizes a randomized controlled experimental design to examine the effects of different types of AI assistance on managerial decision-making and theory development. Participants will be randomly assigned to one of six experimental conditions, varying by the type of AI assistance (none, general AI, specialized AI) and the plausibility of the decision-making problem (plausible vs. implausible).

Participants

Recruitment:

Participants will be recruited via Prolific, focusing on individuals with managerial experience. Efforts will be made to achieve balanced gender representation and diversity across educational backgrounds and industries.

Participants will be randomly assigned to one of six experimental groups:

Plausible Challenge

Control – Single Hypothesis (Group 1a): No Additional Assistance

Control – Multiple Hypotheses (Group 1b): No Additional Assistance

General AI Assistance – Single Hypothesis (Group 2a): Access to ChatGPT

General AI Assistance – Multiple Hypotheses (Group 2b): Access to ChatGPT

Specialized Agentic AI Assistance – Single Hypothesis (Group 3a): Access to agentic AI (TBV-trained AI)

Specialized Agentic AI Assistance – Multiple Hypotheses (Group 3b): Access to agentic AI (TBV-trained AI)

Implausible Challenge

Control – Single Hypothesis (Group 4a): No Additional Assistance

Control – Multiple Hypotheses (Group 4b): No Additional Assistance

General AI Assistance – Single Hypothesis (Group 5a): Access to ChatGPT

General AI Assistance – Multiple Hypotheses (Group 5b): Access to ChatGPT

Specialized Agentic AI Assistance – Single Hypothesis (Group 6a): Access to agentic AI (TBV-trained AI)

Specialized Agentic AI Assistance – Multiple Hypotheses (Group 6b): Access to agentic AI (TBV-trained AI)

Procedure

All participants will first consent to participate and complete a baseline survey gathering demographic data (age, gender, education level, education field, industry, job function, experience, managerial experience).

Introduction Video:

Participants will watch an introductory video covering

The Theory-Based View (TBV)

The role of theories in strategic decision-making.

How theories contribute to value creation and capture.

Decision Problem Presentation:

Participants will be presented with either a plausible or implausible decision-making problem, based on their assigned group, designed to stimulate strategic thinking.

Group-Specific Interventions:

Plausible Challenge

Groups 1a & 1b (Control - Human Only):

1a (Single hypothesis): Participants generate one theory without any AI assistance.

1b (Multiple hypotheses): Participants generate multiple theories without any AI assistance.

Objective: Establish a baseline for theory quality, novelty, and confidence.

Groups 2a & 2b (General AI - ChatGPT):

2a (Single hypothesis): Participants generate one theory with ChatGPT assistance.

2b (Multiple hypotheses): Participants generate multiple theories with ChatGPT assistance.

Objective: Assess the effect of general AI assistance on theory development quality, novelty, and confidence.

Groups 3a & 3b (Specialized Agentic AI - Aristotle):

3a (Single hypothesis): Participants generate one theory with specialized Aristotle AI assistance.

3b (Multiple hypotheses): Participants generate multiple theories, with the option to autogenerate theories using Aristotle.

Objective: Evaluate the enhanced impact of specialized AI on theory quality, novelty, and confidence.

Implausible Challenge

Groups 4a & 4b (Control - Human Only):

4a (Single hypothesis): Participants generate one theory without any AI assistance.

4b (Multiple hypotheses): Participants generate multiple theories without any AI assistance.

Objective: Baseline measurement for challenging decision scenarios.

Groups 5a & 5b (General AI - ChatGPT):

5a (Single hypothesis): Participants generate one theory with ChatGPT assistance.

5b (Multiple hypotheses): Participants generate multiple theories with ChatGPT assistance.

Objective: Investigate general AI’s effect on theory development quality, novelty, and confidence in challenging scenarios.

Groups 6a & 6b (Specialized Agentic AI - Aristotle):

6a (Single hypothesis): Participants generate one theory with Aristotle AI assistance.

6b (Multiple hypotheses): Participants generate multiple theories, including autogenerated options from Aristotle AI.

Objective: Determine the specialized AI's capacity to enhance theory quality, novelty, and confidence in complex decision-making contexts.

Theory Development Task:

Participants will:

Reflect on the given decision problem.

Generate and develop their theories using the provided assistance.

Submit detailed explanations of their approaches and solutions.

Post-Treatment Questionnaire 1:

Participants will again assess:

Confidence: Level of confidence demonstrated in developed theories.

Expected Probability of Success (π): Evaluated based on theoretical explanations and justifications provided by participants.

Self-evaluation of the 5 quality measures developed by Lakhani et al. 2024

Post-Treatment Questionnaire 2:

Participants will answer questions regarding:

Some of the question in the baseline survey, including AI aversion, and GPT familiarity

Knowledge Breadth

Knowledge Depth

Awareness of the Unknown (Unforeseen contingencies)

Participants will receive a debrief that includes:

An explanation of the study's purpose.

The nature of the experimental conditions.

Information about how their data will be used.

Contact information for any follow-up questions.

Evaluation

A specially instructed AI agents, blind to treatment conditions, will evaluate submissions on:

Quality:

Logical coherence (on the reasoning elements and logical reasoning)

Strategic relevance and practical applicability, using the 5 quality measures developed by Lakhani et al. 2024

Expected Probability of Success (π) and the confidence on it:

Evaluated based on theoretical explanations and justifications provided by participants.

And we count the number of theories the user enters in the experiment.

Design details

Randomization method
To allocate participants to experimental conditions, we use minimized blocked randomization (batch size = 50) based on three covariates: a) education level and field; b) AI aversion; and c) level of self-confidence. This provides the baseline allocation of participants to the experimental conditions. To ensure a balanced number of participants in each group and to manage the flow of the experiment, we limit the number of participants who can enter the experimental platform at the same time. This helps prevent an imbalance at the end of the experiment caused by participants who start but do not finish the experiment.
Randomization unit
Individuals
Clusters
No clusters
Observations
2500 managers from Prolific
Arms
208
Primary outcomes
Main Outcomes: Expected Success Probability (π) Mechanism Variables: Number of Hypotheses (n) Theory Quality (z) Confidence in Theory (ω) Confidence in π
Secondary outcomes
Might include other proxies for the potential plausibility and value of the solutions to the decision challenge, such as: * Novelty * Strategic Viability * Financial Value * Environmental Value * Quality of each solution based on the previous 4

Cited by

No citation data available for this registration yet.

Keywords