Upskilling in Data Science with GenAI: How far can we go?

How to cite

Abbadi, Mohamed; Krayer, Lisa; Sack, Dan; Candelon, Francios; Kennedy, Ryan; Awasthi, Urvi; Arnolds, Cristian; Lukic, Vladimir. 2024. "Upskilling in Data Science with GenAI: How far can we go?." AEA RCT Registry. AEARCTR-0013123. https://doi.org/10.1257/rct.13123-2.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Mohamed Abbadi* , Lisa Krayer , Dan Sack , Francios Candelon , Ryan Kennedy , Urvi Awasthi , Cristian Arnolds , Vladimir Lukic

* Primary investigator

Registry ID
AEARCTR-0013123
DOI
https://doi.org/10.1257/rct.13123-2.1
First registered
Last updated
Status
In development
Trial dates
2024-03-15 – 2025-03-08
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/13123

Abstract

Generative Artificial Intelligence (GenAI) is revolutionizing the workplace in ways we have never seen before. In a recent study, researchers show how GenAI could improve productivity of workers and quality of output in tasks that are within GenAI capabilities (Dell'Acqua, et al., 2023). In this study, we aim to explore a new facet of human - GenAI interaction, exploring the potential of using GenAI chatbots to upskill participants in a domain with which the participants are not familiar. We will use Data Science as the test case because this is within the domain of expertise of ChatGPT/GPT-4 with Code Interpreter, which will be the tool used in the experiment. Moreover, we seek to identify areas where GenAI might fall short in its ability to act as an effective coach and where participants using ChatGPT will be unable to substitute for data science domain knowledge and expertise.

Intervention

We plan to give participants in the treatment group access to ChatGPT-4 with basic training on how to use it (less than 30 minutes). Participants in the control group will have access to general search tools (e.g., Google) but won't have to ChatGPT

Experimental design

We will conduct an online randomized control trial to assess the impact of GenAI on upskilling in data science. Initially, we will invite associates, and consultants from a large global consulting firm to join a GenAI study via email. Associates and consultants who register will be randomly assigned to either a Control or a ChatGPT experimental condition group. In parallel, we plan to recruit data scientists to participate in a similar exercise, serving as a benchmark for the typical performance of a data scientist.
After consent, each participant will fill out a pre-survey, go over a short training session depending on experimental condition, complete 2 out of 3 data science tasks (randomly assigned) - with or without the help of ChatGPT depending on experimental condition, and fill out a post survey.

Design details

Randomization method
We used computer-generated numbers to stratify and randomly assign our participants to a control or treatment arm.
Randomization unit
Individual
Clusters
No clustering
Observations
Around 9,00 participants
Arms
Around 900
Primary outcomes
We measure the impact of intervention on two types of outcomes – task-related and self-reported outcomes. On the task-related outcomes, we will measure the impact of ChatGPT on the level of accuracy (correctness and approach) and time taken to complete three data science tasks (problem solving, statistical knowledge and coding) by associates and consultants compared to their counterparts in the control group without ChatGPT. On the task-related outcomes, we will benchmark outcomes of both the control and the treatment to data scientists that do not have access to ChatGPT but have access to general search engines (e.g., Google). On the self-reported outcomes, we will measure the impact of the intervention on their professional identity, beliefs about their career development and ability to complete data-science tasks. We will compare self-reported outcomes pre- and post-survey and across control and treatment groups.
Secondary outcomes
In addition to the primary outcomes, we plan to explore various factors that may influence the performance and outcomes of associates and consultants. These factors include grit, mental presence, and background in STEM or other data science-related disciplines such as statistics, mathematics, and economics.

Cited by

No citation data available for this registration yet.

Keywords