Generative AI & TeamWork: An experimental approach.

How to cite

Dell'Acqua, Fabrizio; Ayoubi, Charles; Lakhani, Karim. 2024. "Generative AI & TeamWork: An experimental approach.." AEA RCT Registry. AEARCTR-0013603. https://doi.org/10.1257/rct.13603-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Fabrizio Dell'Acqua* , Charles Ayoubi , Karim Lakhani

* Primary investigator

Registry ID
AEARCTR-0013603
DOI
https://doi.org/10.1257/rct.13603-1.1
First registered
Last updated
Status
On going
Trial dates
2024-05-13 – 2024-07-25
Countries
Private
JEL codes
C93, D81
Registry record
http://www.socialscienceregistry.org/trials/13603

Abstract

This study examines the effects of integrating a Generative AI (GenAI) tool (a company interface based on GPT4), on team dynamics and the division of labor in a large multinational consumer product company. Participants from Commercial and R&D functions will be randomly assigned to control and treatment groups to collaborate on real business problems during a one-day virtual hackathon. The control groups include individual controls (not using AI) and a Commercial+R&D team without AI. The treatment groups will have either Commercial or R&D paired with GenAI (T1), or both functions working together with GenAI (T2). Participants will work on compatible tasks using real background data from the Fabric Care, FemCare, Grooming, and Baby Care business units, with the goal of producing detailed one-page business solutions. The study aims to answer research questions on whether GPT-4 can substitute certain functions, change teamwork dynamics, and blur boundaries across functions. Crucially, it allows us to compare teams using AI and teams not using AI. The experiment will be conducted in three sessions across the US and Europe on May 13th-15th. The findings will provide insights into how GenAI tools like GenAI can be effectively integrated into collaborative team settings in knowledge-intensive domains within a multinational corporation.

Intervention

1. The introduction of AI, a Generative AI (GenAI) tool, to specific team compositions:
- T1: Human Employee + AI
- T2: Both Commercial and R&D functions work together with AI.

2. The assignment of participants to different team compositions:
- Control groups:
- Individual controls (not using AI)
- Commercial + R&D team without AI
- Treatment groups:
- T1: Human Employee + AI
- T2: Commercial + R&D + AI

Within each group, team members will be randomized. One team member will be designated as the leader, responsible for submitting the solution and leading the usage of AI by sharing their screen. The other team member will follow and contribute to the task. Teams will work remotely and transcribe their conversations during the hackathon.

After completing their initial task, participants in the control groups (both individual and team) will undergo GenAI training. They will then re-do the task, allowing for a within-participant comparison of performance before and after the training.

The interventions aim to study the effects of integrating AI on team dynamics, division of labor, and the potential substitution of certain functions within the context of a large multinational consumer product company. Participants will collaborate on real business problems using actual background data from various business units (Fabric Care, FemCare, Grooming, and Baby Care) during a one-day virtual hackathon.

Experimental design

This study employs a randomized controlled trial design to investigate the effects of integrating Generative AI (GenAI) on team dynamics, division of labor, and performance in a large multinational consumer product company. Participants from Commercial and R&D functions will be randomly assigned to control and treatment groups to collaborate on real business problems during a one-day virtual hackathon.

Groups:
- Control groups:
- Individual controls (not using AI)
- Commercial + R&D team without AI
- Treatment groups:
- T1: Human Employee + AI
- T2: Commercial + R&D + AI

Design details

Randomization method
Random assignment of participants to the intervention groups is achieved through a computerized random number generator, ensuring that the allocation is both random and concealed until the point of assignment.
Randomization unit
The randomization happens within each unit/geography (Europe/Americas). We have 4 business units and 2 geographies, and thus 8 randomization clusters. The probability of assignment is equal across clusters. 1. Individual level randomization: - Participants are randomly assigned to either the individual control group (not using AI) or one of the team-based groups (Commercial + R&D team without AI, T1: Human Employee + AI, or T2: Commercial + R&D + AI). 2. Within-team randomization: - For the team-based groups (Commercial + R&D team without AI, T1: Human Employee + AI, and T2: Commercial + R&D + AI), team members are randomized within each group. - One team member is designated as the leader, responsible for submitting the solution and leading the usage of AI (if applicable).
Clusters
We are randomizing 852 participants. Additionally, we are including additional participants (participants in band 4+) who are assigned to treated individual group. Finally, participants coming in last minute will be accepted in the control individual group. We will use the additional participants for exploratory analyses.
Observations
We are randomizing 852 participants. Additionally, we are including additional participants (participants in band 4+) who are assigned to treated individual group. Finally, participants coming in last minute will be accepted in the control individual group. We will use the additional participants for exploratory analyses.
Arms
We are randomizing 852 participants. 364 are assigned to the paired control treatment (182 groups); 364 to the paired treatment (182 groups), 94 to the treated singleton; 30 to the control singleton. Note that if one of the two paired members of a group does not show up, the other group member is automatically reassigned to the relevant singleton condition (either control or treatment based on the group assignment)
Primary outcomes
Hackathon output. Evaluations of the hackathon output. Additionally, we are building outcomes from text: GenAI prompts and responses: Collection of all the prompts given to the AI and the corresponding responses generated by the AI during the hackathon. Transcripts of group interactions: Collection of the transcribed conversations among team members during the hackathon.
Secondary outcomes
We will measure completion rates and timing for completing the task. We are collecting survey question responses (see the Qualtrics flow). Additionally, we are building outcomes from text: GenAI prompts and responses: Collection of all the prompts given to the AI and the corresponding responses generated by the AI during the hackathon. Transcripts of group interactions: Collection of the transcribed conversations among team members during the hackathon.

Cited by

No citation data available for this registration yet.

Keywords