The Adoption of Generative Artificial Intelligence

How to cite

Humlum, Anders; Vestergaard, Emilie. 2023. "The Adoption of Generative Artificial Intelligence." AEA RCT Registry. AEARCTR-0012527. https://doi.org/10.1257/rct.12527-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Anders Humlum* , Emilie Vestergaard

* Primary investigator

Registry ID
AEARCTR-0012527
DOI
https://doi.org/10.1257/rct.12527-1.0
First registered
Last updated
Status
In development
Trial dates
2023-11-17 – 2024-12-31
Countries
Denmark ()
Registry record
http://www.socialscienceregistry.org/trials/12527

Abstract

The arrival of ChatGPT marks the era of Generative Artificial Intelligence (AI), in which intelligent algorithms create new text, images, and other media. ChatGPT is predicted to boost productivity in several occupations (Eloundou et al., 2023), especially among workers with less expertise in the exposed occupations (Noy & Zhang, 2023).

This project asks three questions: Which workers have adopted ChatGPT? What do workers believe about the capabilities of ChatGPT in their job tasks? Do worker beliefs about the capabilities of ChatGPT determine their adoption?

To answer these questions, we conduct a survey with an embedded randomized treatment among Danish workers in 11 occupations exposed to ChatGPT. We link the survey to registry data on the characteristics and outcomes of workers, including their occupation, skill, experience, workplace, and earnings, among others. We build the survey around the task model of technology adoption (Acemoglu & Autor, 2011) and use the Occupational Information Network (O*NET) database to identify the relevant job tasks for each occupation.

We conduct a main survey and a follow-up.

The main survey is organized into five parts. First, we ask workers about their experiences with ChatGPT in their job tasks. We also ask about the importance of these tasks for workers and their expertise in the tasks. Second, we elicit workers' beliefs about the productivity and expertise-complementarity of ChatGPT in their job tasks. Third, we expose our treatment groups to expert assessments of the capabilities of ChatGPT (productivity and expertise-complementarity) in the job tasks. Fourth, we ask workers about their intended use of ChatGPT in their job tasks. We also allow workers to receive additional information on ChatGPT in their occupations. Finally, we ask workers about their beliefs about the productivity of ChatGPT in their own jobs, together with any misalignments in their stated beliefs and intended behaviors.

We conduct a follow-up survey two weeks after the workers' response to the main survey. The follow-up measures whether workers' beliefs persist over time and if workers' intended adoption translates into the actual use of ChatGPT.

We randomize the sampling and treatment by workers and workplaces. The adoption study described in the analysis plan focuses on the worker-level experiment.

Intervention

We will run a survey with a randomized information provision experiment on the capabilities of ChatGPT in workers’ job tasks.

Treatment 1 ("productivity") exposes workers to information about the general productivity of ChatGPT in each job task. Following Eloundou et al. (2023), we inform workers whether access to ChatGPT can reduce the time it takes for an average worker to complete a task by at least half (50%).

Treatment 2 ("complementarity") exposes workers to information about whether the time savings from ChatGPT are smaller, similar, or larger for workers with expertise in that task.

The treatment groups see a comparison of their assessments and the expert assessments, together with brief explanations of the expert assessments. The control group only sees a summary of their prior assessments.

Experimental design

Our experiment exposes workers to expert assessment of the capabilities of ChatGPT in different job tasks. Our expert assessments are based on Eloundou et al. (2023), adapted to the Danish context, and validated by industry specialists.

We have three treatment arms:

Treatment 1 ("productivity"): Workers receive information about the general productivity of ChatGPT in each job task. Following Eloundou et al. (2023), we inform workers whether access to ChatGPT can reduce the time it takes for an average worker to complete a task by at least half (50%).

Treatment 2 ("complementarity"): Workers receive information about whether the time savings from ChatGPT are smaller, similar, or larger for workers with expertise in that task.

Control: The control group only sees a summary of their prior assessments.

The treatment groups see a comparison of their assessments and the expert assessments, together with brief explanations of the expert assessments.

Prior to the treatment, we ask workers about the importance of the job tasks and their expertise in the tasks. We also ask workers about their experiences with ChatGPT in the job tasks. Furthermore, we elicit workers' prior beliefs about the general productivity (treatment 1) and expertise-complementarity (treatment 2) of ChatGPT in the job tasks.

After the treatment, we ask workers about their intended use of ChatGPT in their job tasks. We also allow workers to sign up for additional material on potential use cases of ChatGPT in their occupations.

Finally, we ask workers about their beliefs about the productivity of ChatGPT in their own jobs. We also ask workers about misalignments in their stated beliefs and intended behaviors. The follow-up questions unpack the chain from information provision via updated beliefs to adoption behaviors.

We will conduct a follow-up survey two weeks after the workers' response to the main survey. The follow-up will assess (i) whether workers' beliefs about ChatGPT persist over time, (ii) whether workers' intended adoption of ChatGPT manifests into the actual adoption, and (iii) short-run spillovers across coworkers in belief formation and adoption behaviors.

Additional information on selecting occupations:
We survey occupations that meet the following criteria:
1. The occupation has a well-identified occupational code (ISCO and SOC).
2. The occupation has at least one job task in which ChatGPT can reduce the time it takes for an average worker to complete it by at least half (Eloundou et al., 2023).
3. The occupation has enough workers for statistical analysis.

The resulting list of occupations is:
Accountants
Customer supporters
Financial advisors
HR professionals
IT supporters
Journalists
Legal professionals
Marketing professionals
Software developers
Teachers
Office clerks

Additional information on selecting job tasks:
We use the Occupational Information Network (O*NET) to identify the job tasks for each occupation. For each occupation, we select 6 job tasks that represent the capabilities of ChatGPT ("general productivity" and "expertise-complementarity") in the occupation. For task combinations with similar representation, we prioritize combinations with higher task relevances, as measured by O*NET.

Design details

Randomization method
We randomize sampling and treatment using a two-stage randomization procedure by workplaces and workers. In the first stage, we split workplaces into strata based on sampling and treatment intensities. In the second stage, we sample workers and assign them treatments based on their workplace-specific sampling and treatment intensities. We use Statas random-number generator (“runiform”) with a pre-specified seed. Sampling: We randomly allocate workplaces into three strata with sampling rates of 0%, 50%, and 100% (with an equal number of workplaces in strata 2 and 3). Workers are sampled randomly according to their workplace-specific sampling rates. Treatment arms: We divide workplaces into four strata with the following shares of (treatment 1, treatment 2, and control) workers: (2/9, 2/9, 5/9), (2/9, 4/9, 3/9), (4/9, 2/9, 3/9), (4/9, 4/9, 1/9). Sampled workers are randomly allocated to treatment arms based on their workplace-specific treatment rates. Prizes: Our survey includes randomized participation incentives. In particular, we use simple randomization to allocate respondents into four prize categories (1000, 2500, 5000, and 10000 DKK) with equal shares.
Randomization unit
We randomize by workplaces and workers.
Clusters
We randomly allocate workplaces into three strata with sampling rates of 0%, 50%, and 100% (with an equal number of workplaces in strata 2 and 3). We divide workplaces into four strata with the following shares of (treatment 1, treatment 2, and control) workers: (2/9, 2/9, 5/9), (2/9, 4/9, 3/9), (4/9, 2/9, 3/9), (4/9, 4/9, 1/9). Please see the attached note “Sampling and Randomization” for details on the number of workplaces in the strata.
Observations
We will contact a total of 115,000 individuals in November-January 2023. We first invite 100,000 individuals to our main survey. All respondents of the main survey are invited to the follow-up. In addition, we invite 15,000 individuals only to the follow-up. We split the sample equally among the 11 occupations listed above. We use register data to identify the occupations of individuals. For occupations with fewer workers than the required sample, we distribute the excess sample equally among the remaining occupations. Please see the attached note “Sampling and Randomization” for details on the number of sampled individuals by occupation.
Arms
Please see the attached note “Sampling and Randomization” for details on the number of workplaces and workers by treatment arms.
Primary outcomes
1. Beliefs about the capabilities of ChatGPT in workers' job tasks (productivity and expertise-complementarity). 2. Adoption of ChatGPT (intended and actual).

Cited by

No citation data available for this registration yet.

Keywords