The Predictive Validity of Audio and Text Responses to Open-Ended Survey Questions for Latent Individual Traits
How to cite
Galasso, Vincenzo; Nannicini, Tommaso. 2025. "The Predictive Validity of Audio and Text Responses to Open-Ended Survey Questions for Latent Individual Traits." AEA RCT Registry. AEARCTR-0016672. https://doi.org/10.1257/rct.16672-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Understanding individuals’ beliefs, preferences, and motivations is essential in social sciences. Recent technological advancements—notably, large language models (LLMs) for analyzing openended responses and the diffusion of voice messaging— have the potential to significantly
enhance our ability to elicit these dimensions. Previous studies (Galasso et al., 2024) have investigated the differences between oral
and written responses to open-ended survey questions. Using a series of randomized controlled trials across three surveys (focused on AI, public policy, and international relations), Galasso et al., (2024) showed that respondents who provided audio answers gave longer, though lexically simpler, responses, offering more information and containing more personal experiences than written responses. These findings thus suggest that oral responses to open-ended questions can capture richer, more personal insights, presenting a valuable method for understanding individual reasoning.
In this study, we propose two extensions:
1. Editing Opportunity: Providing respondents with the transcript of their audio (or text) answer and allowing them to revise the answer may reduce oversharing and personal content.
2. Predictive Validity: Audio-based open-ended answers may better predict latent individual characteristics—here, attitudes and behaviors toward immigrants—than text responses alone.
We also explore how respondents’ Big Five traits relate to the informativeness and subjectivity of open-ended answers across modes to test whether more extroverted people are more likely to provide more, and more personal, information when recording an audio.
Intervention
Participants will be randomly assigned to one of four experimental conditions that vary the mode in which they provide answers to open-ended survey questions:
Audio: Respondents answer all open-ended questions orally (recorded audio).
Text: Respondents answer all open-ended questions in written form.
Audio with Transcript & Prompt to Modify: Respondents first answer all open-ended questions orally. They are then shown a transcript of their spoken response and asked whether they wish to revise or modify it in writing.
Text with Transcript & Prompt to Modify: Respondents first answer all open-ended questions in written form. They are then shown the text of their response again and asked whether they wish to revise or modify it in writing.
This design allows us to isolate the effects of response modality (audio vs. text) and the additional opportunity to review and edit responses (transcript & prompt to modify).
Experimental design
Participants will be randomly assigned to one of four experimental conditions that vary the mode in which they provide answers to open-ended survey questions:
Audio: Respondents answer all open-ended questions orally (recorded audio).
Text: Respondents answer all open-ended questions in written form.
Audio with Transcript & Prompt to Modify: Respondents first answer all open-ended questions orally. They are then shown a transcript of their spoken response and asked whether they wish to revise or modify it in writing.
Text with Transcript & Prompt to Modify: Respondents first answer all open-ended questions in written form. They are then shown the text of their response again and asked whether they wish to revise or modify it in writing.
Design details
Cited by
No citation data available for this registration yet.