Conducting Qualitative Interviews with AI

How to cite

Chopra, Felix; Haaland, Ingar. 2024. "Conducting Qualitative Interviews with AI." AEA RCT Registry. AEARCTR-0015069. https://doi.org/10.1257/rct.15069-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Felix Chopra* , Ingar Haaland

* Primary investigator

Registry ID
AEARCTR-0015069
DOI
https://doi.org/10.1257/rct.15069-1.0
First registered
Last updated
Status
On going
Trial dates
2023-08-22 – 2026-09-30
Countries
Private
JEL codes
C83, C90, D14, D91, Z13
Registry record
http://www.socialscienceregistry.org/trials/15069

Abstract

We conduct a comprehensive examination of the quality of AI-led interviews using a combination. For this purpose, we conducted qualitative interviews with human subjects by delegating the task of interviewing to an AI. We show that the AI interviewer follows methodological guidelines and engages participants. AI-led interviews allow hypothesis generation, outperform standard open-ended survey questions by broadening the set of measurable phenomena, and predict economic behavior half a year later. We conduct additional surveys and experiments to complement these findings.

Intervention

We conducted a series of data collections to study the quality and feasibility of AI-led interviews. Studies 1, 2, 3 and 5 do not include an experimental manipulation. Study 4 is an experiment.

1) Main study (n=381, Aug-Sep 2023): Descriptive survey with an integrated 30-minute text-based interview led by an AI interviewer. Post-interview survey questions about the interview experiences. Additional survey modules on factors associated with stock market non-participation, respondent characteristics, and household finances.

2) Follow-up study (n=266, May 2024): Long-term follow-up with participants from the main study. Descriptive survey that includes incentivized outcomes to study whether selected interview codes predict choices: (i) demand for insurance against nominal losses, (ii) willingness to pay for an investment course, (iii) incentivized belief about the performance of actively managed funds.

3) Validation survey (n=1,000, May 2024): Descriptive survey with an equal share of stockowners and non-owners. Survey questions aimed at measuring whether respondents hold the "active investing" mental model hypothesized based on the qualitative analysis of the interview transcripts from the main study.

4) Selection experiment (n=499, Nov 2023): Respondents are asked to select all studies they would like to participate in if we decide to run it: (i) 40-minute survey, (ii) 40-minute interview with a human interviewer, (iii) 40-minute interview with an AI interviewer. These studies are shown on separate pages in randomized order (between subjects). The main outcome of interest is the revealed demand for study participation from a between-subject comparison of the first vignette (binary). We additionally explore factors predicting selection into AI-led interviews using within-subject variation.

5) Multiple open-ended questions benchmark (n=384, Nov 2024): Respondents are asked to answer six pre-determined open-ended survey questions about their reasons for stock market non-participation. The first question is identical to the opening question of our AI-led interviews; the other open-ended questions mimic the topic guide of our interviews.

Experimental design

Study 4 ("Selection experiment", n=499, Nov 2023): We recruit respondents from Prolific for a survey experiment. As part of the survey, we ask respondents to indicate for three separate studies whether they would like to participate in them (yes/no). The studies are: (i) 40-minute survey, (ii) 40-minute interview with a human interviewer, (iii) 40-minute interview with an AI interviewer. These studies are shown on separate pages. The treatment is the order in which the studies are shown, which we randomize at the respondent level. The main outcome of interest is the share of respondents selecting that they would like to participate in a given study. We compare the demand for AI-led interviews with the demand for participating other studies (e.g. human-led or survey) using the between subject variation from the first vignette. We are testing whether presenting a study as including an "AI interview" affects the demand for participating in the study. We additionally explore factors predicting selection into AI-led interviews using the within-subject variation.

The other studies do not include an experimental intervention.

Design details

Randomization method
Randomization is done by the survey software (e.g. Qualtrics).
Randomization unit
The randomization unit is a survey respondent.
Clusters
see above
Observations
see above
Arms
see above
Primary outcomes
see above

Cited by

No citation data available for this registration yet.

Keywords