Upskilling in Data Science with GenAI: How far can we go?
How to cite
Abbadi, Mohamed; Krayer, Lisa; Sack, Dan; Candelon, Francios; Kennedy, Ryan; Awasthi, Urvi; Arnolds, Cristian; Lukic, Vladimir. 2024. "Upskilling in Data Science with GenAI: How far can we go?." AEA RCT Registry. AEARCTR-0013123. https://doi.org/10.1257/rct.13123-2.1
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Generative Artificial Intelligence (GenAI) is revolutionizing the workplace in ways we have never seen before. In a recent study, researchers show how GenAI could improve productivity of workers and quality of output in tasks that are within GenAI capabilities (Dell'Acqua, et al., 2023). In this study, we aim to explore a new facet of human - GenAI interaction, exploring the potential of using GenAI chatbots to upskill participants in a domain with which the participants are not familiar. We will use Data Science as the test case because this is within the domain of expertise of ChatGPT/GPT-4 with Code Interpreter, which will be the tool used in the experiment. Moreover, we seek to identify areas where GenAI might fall short in its ability to act as an effective coach and where participants using ChatGPT will be unable to substitute for data science domain knowledge and expertise.
Intervention
We plan to give participants in the treatment group access to ChatGPT-4 with basic training on how to use it (less than 30 minutes). Participants in the control group will have access to general search tools (e.g., Google) but won't have to ChatGPT
Experimental design
We will conduct an online randomized control trial to assess the impact of GenAI on upskilling in data science. Initially, we will invite associates, and consultants from a large global consulting firm to join a GenAI study via email. Associates and consultants who register will be randomly assigned to either a Control or a ChatGPT experimental condition group. In parallel, we plan to recruit data scientists to participate in a similar exercise, serving as a benchmark for the typical performance of a data scientist.
After consent, each participant will fill out a pre-survey, go over a short training session depending on experimental condition, complete 2 out of 3 data science tasks (randomly assigned) - with or without the help of ChatGPT depending on experimental condition, and fill out a post survey.
Design details
Cited by
No citation data available for this registration yet.