Statistical Literacy in the Newsroom: Experimental Evidence

How to cite

Kerkhof, Anna; Noske, Nikola; Berger, Lara Marie. 2024. "Statistical Literacy in the Newsroom: Experimental Evidence." AEA RCT Registry. AEARCTR-0014037. https://doi.org/10.1257/rct.14037-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Anna Kerkhof* , Nikola Noske , Lara Marie Berger

* Primary investigator

Registry ID
AEARCTR-0014037
DOI
https://doi.org/10.1257/rct.14037-1.0
First registered
Last updated
Status
In development
Trial dates
2024-07-23 – 2024-08-31
Countries
Private
JEL codes
A20, H40, I20, L82, L86
Registry record
http://www.socialscienceregistry.org/trials/14037

Abstract

Quality journalism can make scientific publications and statistics accessible for the general public. Yet, numerous studies reveal that many journalists lack proficiency in statistical analysis. To improve the statistical literacy of journalists, we have developed a concise educational video focusing on the accurate interpretation of scientific studies and statistics. The video addresses key topics such as (random) sampling, computation and interpretation of probabilities, differences between correlation and causation, and identification of misleading figures.
To assess the efficacy of the video, we recruit trained journalists through a mailing list of the German Journalistic Association. These journalists are randomly assigned to either the treatment or a control group (between-subject design). The treatment group receives the video immediately, the control group only after the experiment. All participants are asked to assess several news articles on scientific studies in terms of their accuracy. Computing differences in means between treatment and control group gives an estimate about the efficacy of the video.

Intervention

Educational video (see attachment)

Experimental design

Survey flow:
We invite around 300 journalists via a mailing list of the German Journalistic Association and ask if they are interested in participating in our study. Those who are interested will be randomly assigned to the treatment group or the control group. Journalists in the treatment group see our educational video (see attachment). Journalists in the control group will see the video after the trial.
Next, we ask all participants to read two news articles covering scientific studies. We also offer them to read the original scientific studies through a hyperlink. For each participant, we randomly draw two studies (plus two corresponding news articles, i.e., four news articles in total) from a pool of five studies. We display only the most relevant parts of the news articles (e.g., title, teaser, beginning), but the entire news articles are available through drop-down menues. For all news articles, the authors have pre-assessed the type and number of reporting mistakes (if any), following a codebook (see attachment). The pre-assessment is pre-registered here (see attachment), but will not be visible to our participants.
Then, for each news article, we ask our participants to assess whether there are mistakes in the reporting or not (e.g., misinterpretation of statistics), how many mistakes there are, and which types of mistakes (see "primary outcomes"). Then we ask them to suggest a hypothetical alternative title for an article covering the respective study (see "secondary outcomes").
After the trial, journalists in the control group receive access to the educational video, too.

Analysis:
In our main analysis, we will compare the treatment to the control group through simple OLS regressions with each of our outcomes on the LHS, respectively, and a dummy for being in the treatment group (or not) on the RHS. The parameter that measures the impact of the treatment indicator is our parameter of interest. Control variables include gender, age, and political orientation.

Design details

Randomization method
randomization through computer
Randomization unit
individual journalists
Clusters
300 journalists, Note: If we no not reach the planned number of observations, we will close the survey two weeks after sending out the invitations.
Observations
300 journalists
Arms
150 journalists per treatment group
Primary outcomes
1) Presence of reporting errors in the articles on scientific studies 2) Absolute number of errors per article 3) Type of errors: i) correlation vs. causation, ii) lack of context, iii) misinterpretation of statistics, iv) sample not representative /selection, v) other
Secondary outcomes
Individual title

Cited by

No citation data available for this registration yet.

Keywords