The Predictive Validity of Audio and Text Responses to Open-Ended Survey Questions for Latent Individual Traits

How to cite

Galasso, Vincenzo; Nannicini, Tommaso. 2025. "The Predictive Validity of Audio and Text Responses to Open-Ended Survey Questions for Latent Individual Traits." AEA RCT Registry. AEARCTR-0016672. https://doi.org/10.1257/rct.16672-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Vincenzo Galasso* , Tommaso Nannicini

* Primary investigator

Registry ID
AEARCTR-0016672
DOI
https://doi.org/10.1257/rct.16672-1.0
First registered
Last updated
Status
In development
Trial dates
2025-10-09 – 2025-11-30
Countries
Private
JEL codes
C83, D83
Registry record
http://www.socialscienceregistry.org/trials/16672

Abstract

Understanding individuals’ beliefs, preferences, and motivations is essential in social sciences. Recent technological advancements—notably, large language models (LLMs) for analyzing openended responses and the diffusion of voice messaging— have the potential to significantly
enhance our ability to elicit these dimensions. Previous studies (Galasso et al., 2024) have investigated the differences between oral
and written responses to open-ended survey questions. Using a series of randomized controlled trials across three surveys (focused on AI, public policy, and international relations), Galasso et al., (2024) showed that respondents who provided audio answers gave longer, though lexically simpler, responses, offering more information and containing more personal experiences than written responses. These findings thus suggest that oral responses to open-ended questions can capture richer, more personal insights, presenting a valuable method for understanding individual reasoning.
In this study, we propose two extensions:
1. Editing Opportunity: Providing respondents with the transcript of their audio (or text) answer and allowing them to revise the answer may reduce oversharing and personal content.
2. Predictive Validity: Audio-based open-ended answers may better predict latent individual characteristics—here, attitudes and behaviors toward immigrants—than text responses alone.
We also explore how respondents’ Big Five traits relate to the informativeness and subjectivity of open-ended answers across modes to test whether more extroverted people are more likely to provide more, and more personal, information when recording an audio.

Intervention

Participants will be randomly assigned to one of four experimental conditions that vary the mode in which they provide answers to open-ended survey questions:
Audio: Respondents answer all open-ended questions orally (recorded audio).
Text: Respondents answer all open-ended questions in written form.
Audio with Transcript & Prompt to Modify: Respondents first answer all open-ended questions orally. They are then shown a transcript of their spoken response and asked whether they wish to revise or modify it in writing.
Text with Transcript & Prompt to Modify: Respondents first answer all open-ended questions in written form. They are then shown the text of their response again and asked whether they wish to revise or modify it in writing.
This design allows us to isolate the effects of response modality (audio vs. text) and the additional opportunity to review and edit responses (transcript & prompt to modify).

Experimental design

Participants will be randomly assigned to one of four experimental conditions that vary the mode in which they provide answers to open-ended survey questions:
Audio: Respondents answer all open-ended questions orally (recorded audio).
Text: Respondents answer all open-ended questions in written form.
Audio with Transcript & Prompt to Modify: Respondents first answer all open-ended questions orally. They are then shown a transcript of their spoken response and asked whether they wish to revise or modify it in writing.
Text with Transcript & Prompt to Modify: Respondents first answer all open-ended questions in written form. They are then shown the text of their response again and asked whether they wish to revise or modify it in writing.

Design details

Randomization method
randomization done by the Survey Company software that will allocate respondents in the four arms.
Randomization unit
Individual level: respondents to the survey
Clusters
For budgetary reasons, the national probability sample will consist of N ≈ 2,000 adult respondents recruited by a professional survey company, with CAWI methodology.
Observations
Observation for almost all outcome will coincide with the number of individual respondents. In the conjoint experiment, each individual will rank 5 pairs of scenarios. Since the observation is a scenario, in the conjoint experiment, there will be 20,000 observations
Arms
500 individuals each
Primary outcomes
From answers to open ended questions, we calculate (i) number of words and of significant words; (ii) Yule’s K and the Type-Token Ratio (TTR) to measure lexical diversity; (iii) informativeness score and informativeness dummy and (iv) use the pronoun “I” (in Italian, “Io”); use of verbs in the first-person singular; report of respondent’s personal experience; and subjectivity.
Secondary outcomes
(i) Average number of items in a List Experiment using Sensitive item on immigrants in a list format; (ii) Allocation of resources to recipients and to NGOs varying by ethnic cue in a Dictator Game; (iii) marginal willingness to pay in a Conjoint Experiment of Residential choice tasks, including immigrant share attribute; (iv) Closed ended comfort rating on immigrant share (1-5 Likert scale).

Cited by

No citation data available for this registration yet.

Keywords