Persistent Stories or Persistent Numbers: Effects of Narrative versus Statistical Evidence on Policy Preferences

How to cite

Nakabayashi, Masaki; Kawata, Keisuke; McElwain, Kenneth. 2022. "Persistent Stories or Persistent Numbers: Effects of Narrative versus Statistical Evidence on Policy Preferences." AEA RCT Registry. AEARCTR-0008821. https://doi.org/10.1257/rct.8821-1.2

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Masaki Nakabayashi* , Keisuke Kawata , Kenneth McElwain

* Primary investigator

Registry ID
AEARCTR-0008821
DOI
https://doi.org/10.1257/rct.8821-1.2
First registered
Last updated
Status
In development
Trial dates
2022-03-07 – 2023-03-22
Countries
Japan ()
JEL codes
D19; H53; I38
Registry record
http://www.socialscienceregistry.org/trials/8821

Abstract

1. Motivation
Many research studies have shown that intervention effects are short-lived. However, we consider that the short life of a treatment effect originates to a substantial degree a uni-dimensional experimental setting, such as requiring greater efforts to save energy or towels, power, or water. In fact, many essential issues of our lives do not involve uni-dimensional decision-making. When a legislature approves a budget for the coming year, it does not necessarily increase or decrease the taxation rate. Instead, it discusses a possible budget vector while viewing the scholar of the vector, or the revenue, as given. Improvement in public policy does not necessarily mean a larger size of the government but instead could mean wiser spending given a constant revenue. Traditional uni-dimensional design implicitly assumes additional effort or tax payments and might discourage long-term improvement in behavior or spending.

To address this issue, we conceive a multidimensional public policy space in which hypothetical policy packages are randomly generated by a fully randomized conjoint design. The policy space contains a possible shrinking of government size as well as spending for public policies. If a respondent wants to keep the size of government constant or even shrink it, she or he still might want to relieve poverty by reallocating spending in other directions. By allowing for such choices, we can accurately identify treatment effects on policy preferences.

Additionally, possibly different impacts of different modes of information have been discussed. We use both statistical evidence and narratives to exchange findings and ideas on a daily decision-making basis. While we implicitly believe that statistical or narrative evidence might be more effective under specific circumstances, empirical results remain mixed. We estimate the possible difference in choices within our hypothetical public policy space.

2. Objective
We investigate (1) whether information provision affects public policy preferences in the long term by a three-wave panel conjoint experiment performed from March 2022 to September 2022 and to February 2023. If we find that information provision does affect policy preferences, we investigate (2) whether narrative or statistical evidence affects policy preferences stronger/longer. Rom among several critical issues, we adopt poverty relief as a focus and study the relative preference for poverty relief compared with other dimensions of public policy; public works, education, public health care, the public pension fund, and fiscal retrenchment.

3. Setting and design
We conduct a panel internet survey by recruiting a nonprobability sample of 15,000 Japanese adults on the first wave in March 2022 and send consecutive surveys to the same respondents in September 2022 and February 2023. In each wave, we implement a fully randomized conjoint experiment that shows 2 hypothetical allocations (indicated by the percentage of increased consumption tax) to 5 dimensions of public policy: 1) public works, 2) education, 3) public health care, 4) the public pension fund, and 5) poverty relief. Any residual is assumed to be spent for government debt redemption. Respondents choose between the 2 hypothetical policy packages in a task. We assign 5 tasks to each respondent at each wave.

4. Treatment
In the first wave of March 2022, we provide the randomly selected respondents with statistical information on the relative poverty rate in Japan or a narrative on the relative poverty in Japan. Statistical evidence is cited from the National Livelihood Survey 2019 conducted by the Ministry of Health, Labour and Welfare, the government of Japan (https://www.mhlw.go.jp/toukei/saikin/hw/k-tyosa/k-tyosa19/index.html).

5. Results
5.1 Persistency of intervention effects
We investigate how long the effects of information treatment on preferences for public policies persist.

5.2 Levels and lengths of different modes of information
We investigate whether either narrative evidence or statistical evidence for relative poverty in Japan affects policy preferences more strongly than the other and whether either of the effects persists longer than the other.

Intervention

1. Information treatment
In the first wave planned in March 2022, we provide the randomly selected respondents with statistical information about the relative poverty rate in Japan or a narrative on the relative poverty in Japan. Statistical evidence is cited from the National Livelihood Survey 2019 conducted by the Ministry of Health, Labour and Welfare, the government of Japan (https://www.mhlw.go.jp/toukei/saikin/hw/k-tyosa/k-tyosa19/index.html).

2. Choice of hypothetical policy packages
We adopt a fully randomized conjoint design to generate 2 5-dimensional public policy package options. Since each level (i.e., budget allocation percentage) of an attribute (i.e., a policy) is selected in forming a policy package, we can identify a change in preference accompanying a change in attribute levels as a causal effect from the latter to the former.

Experimental design

1. Survey design
1. Panel experiment of information treatment effects
We conduct a panel Internet survey by recruiting a nonprobability sample of 15,000 Japanese adults on the first wave in March 2022 and send the consecutive surveys to the same respondent in September 2022 and February 2023. On each wave, we implement a fully randomized conjoint experiment that shows 2 hypothetical allocations indicated by the percentage of raised consumption tax to 5 dimensions of public policies; 1) public works; 2) education; 3) public healthcare; 4) public pension; 5) poverty relief. Any residual is assumed to be spent for government debt redemption. Respondents choose between the 2 hypothetical policy packages at a task. We assign 5 tasks to each respondent on each wave.

2. Information treatment
In the first wave planned in March 2022, we provide the randomly selected respondents with statistical information about the relative poverty rate in Japan or a narrative on the relative poverty in Japan. Statistical evidence is cited from the National Livelihood Survey 2019 conducted by the Ministry of Health, Labour and Welfare, the government of Japan (https://www.mhlw.go.jp/toukei/saikin/hw/k-tyosa/k-tyosa19/index.html).

3. Background characteristics
We survey demographic characteristics, educational backgrounds, income, household income, partisanship, self-conceived right-leaning, preference for the size of government, and self-conceived social status.

Design details

Randomization method
Randomization will be done in a survey company by a computer.
Randomization unit
15,000 respondents.
Clusters
15,000 individuals.
Observations
15,000 individuals.
Arms
5,000 individuals are the control arm and 10,000 individuals are the treatment arm.
Primary outcomes
1. We investigate whether the provision of information regarding poverty affects public policy preference. 2. If we find an effect of information provision mode, we measure how long the effect persists. 3. If we find an effect of information provision, we also investigate whether narrative or statistical evidence has a greater or more persistent impact on policy preference than the other. 4. If either narrative or statistical evidence has a greater or more persistent impact on policy preference than the other, we investigate whether the difference is associated with respondents' choice of the mode of information (i.e., of narrative or statistical evidence).

Cited by

No citation data available for this registration yet.

Keywords