Numbers Tell, but Words Sell: Evidence from Research Communication

How to cite

Thaler, Michael; Toma, Mattie; Wang, Victor Yaneng. 2024. "Numbers Tell, but Words Sell: Evidence from Research Communication." AEA RCT Registry. AEARCTR-0013947. https://doi.org/10.1257/rct.13947-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Michael Thaler* , Mattie Toma , Victor Yaneng Wang

* Primary investigator

Registry ID
AEARCTR-0013947
DOI
https://doi.org/10.1257/rct.13947-1.0
First registered
Last updated
Status
In development
Trial dates
2024-07-16 – 2025-06-30
Countries
Private
JEL codes
D78; D83; D91
Registry record
http://www.socialscienceregistry.org/trials/13947

Abstract

Researchers must often choose whether to communicate using numbers or language when sharing information with policymakers. While there are many reasons why experts would send messages using language or numbers, our study emphasizes the role of message precision: numbers represent precise estimates, whereas language is imprecise because one word can be interpreted as many numerical values. We run an experiment in which we recruit academic researchers and vary their incentives to directionally persuade or to convey accurate information to a sample of policymakers that we recruit. We analyze how researchers choose between message formats, testing whether directional incentives increase their likelihood of using language to communicate as compared to using numeric estimates, and explore these effects on policymakers.

Intervention

For experimental details, see the Experimental Design (Public) section.

Experimental design

Recruitment:

This experiment will be conducted in two waves: The first wave involves "Senders" (recruited from a pool of academic researchers) and the second wave involves "Receivers" (recruited from a pool of policymakers in government using the Warwick Policymakers Lab).

We will recruit senders via emails to academic social-science researchers over the course of two weeks, or until we reach 150 senders, whichever comes first. If we do not reach 50 senders during this time period, we will extend the time period. At the end of the study, we will ask senders if they previously knew about our main hypothesis, and will not include senders who answer "yes" to this question in our main analysis.
We will recruit receivers via participants in the Warwick Policymakers Lab over the subsequent eight weeks, or until we reach 75 Receivers, whichever comes first.

Experiment Timing:

Senders complete multiple rounds. In each round, they learn the treatment effect size of an academic research study. Then, they choose how to communicate this effect size to a participant in the receiver wave as detailed below.

Receivers complete multiple rounds. In each round, they are asked to predict the treatment effect of one of the studies. Then, they are matched with a sender and are given the sender's message. Finally, they are asked to predict the effect size of the study, asked how likely they would be to share the information with their colleagues, and are asked whether they would like to receive an infographic about the study that they can share with their colleagues.

At the end of the study, we will ask subjects a set of questions, including about demographics, which we may use in exploratory analyses.

We will include all completed rounds in our analysis, even in cases when the participant does not finish the survey.

Incentives:

Senders and receivers are paid in Amazon gift cards.

For receivers, one of the questions is randomly selected, and they earn a bonus payment if their prediction of the effect size is sufficiently close to the actual effect size.

For senders, we randomly vary whether their payments are:
- The same as their receiver's ("aligned"), or
- Increasing in their receiver's predictions ("directional").

Next, we describe what messages senders can send. Messages can come in one of two "formats":
- Number messages have the format: "The treatment led to an increase of X percentage points."
- Language messages have the format: "The treatment led to a [word/phrase] increase."

Senders make two choices. They first choose the X they want to communicate for the Number message and the word they want to communicate for the Language message. Then, they choose whether they would rather send their Number or their Language message. One of their two messages will always be the one used, but their preferred format will sometimes not be the one used.

Receivers face these treatments as well. That is, they are either matched with senders who have aligned or directional incentives, but do not know which type of sender they face.

Design details

Randomization method
Randomization for all treatments done by computer.
Randomization unit
We randomize the incentives treatment (aligned or directional) between senders; receivers may face senders with different incentives. There are six studies, and we randomize the studies that participants see.
Clusters
Senders: 150 participants (see notes on recruitment and sample size above) Receivers: 75 participants (see notes on recruitment and sample size above)
Observations
Senders: 900 message choices (450 number messages and 450 language messages), 450 message format choices Receivers: 150 predictions before messages, 150 predictions after messages
Arms
Senders: 1/2 have aligned incentives and 1/2 have directional incentives. Receivers: 1/2 face senders with aligned incentives and 1/2 face senders with directional incentives.
Primary outcomes
H1. We hypothesize that researchers are less likely to choose to send number messages, and thus are more likely to choose to send language messages, when they face a directional incentive.
Secondary outcomes
We may conduct exploratory analyses as well: H2. We hypothesize that senders with directional incentives will be more likely to send number messages for larger (good news) compared to smaller (bad news) effect sizes. Note that this specification will not control for study-level fixed effects. H3. We may explore measuring slant of language and numbers in future waves by benchmarking language to numbers. We hypothesize that senders slant more with directional incentives than with aligned incentives, conditional on their preferred message format. We will also explore whether senders slant more with directional incentives within message formats. H4. We may test whether receivers are persuaded by senders; that is, whether receiver predictions are higher when senders faced directional compared with aligned incentives. Then, we may look at whether senders with directional incentives increase receivers' predictions when language messages are sent relative to when number messages are sent.
Sponsors
Name: British Academy/Leverhulme Trust Location: United Kingdom Url: https://www.thebritishacademy.ac.uk/funding/ba-leverhulme-small-research-grants/

Cited by

No citation data available for this registration yet.

Keywords