Survey experiments on policymakers' perspectives towards evaluation and experimentation

How to cite

Fuller, Rob; Phipps, James; Brackin, Maria. 2025. "Survey experiments on policymakers' perspectives towards evaluation and experimentation." AEA RCT Registry. AEARCTR-0016824. https://doi.org/10.1257/rct.16824-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Rob Fuller* , James Phipps , Maria Brackin

* Primary investigator

Registry ID
AEARCTR-0016824
DOI
https://doi.org/10.1257/rct.16824-1.0
First registered
Last updated
Status
In development
Trial dates
2025-09-19 – 2025-10-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/16824

Abstract

The Innovation Growth Lab (IGL) and Technopolis Group will conduct an online survey of policymakers and policy practitioners across Europe, to assess their current level of understanding of policy experimentation, the drivers of and barriers to use of experimentation, and their views on the potential for a community of practice.

The survey incorporates four randomised experiments:
- A conjoint (discrete-choice) experiment designed to elicit respondents’ willingness to engage in randomised experimentation, and which conditions are conducive to this.
- An A/B test to determine whether respondents’ perceptions of their understanding of experimentation is affected by whether they are prompted with a particularly rigorous definition of what is meant by experimentation.
- An A/B test to examine whether respondents are more supportive of policy experimentation if they are first told of the results of a previous survey in which a majority of the general public in several European countries expressed generally supportive views about experimentation.
- An A/B test to examine whether respondents have different reactions to the terms “randomised experimentation” or “randomised controlled trials (RCTs)”, when asked whether there should be more or less of this type of experiment in research and innovation policy.

Intervention

The interventions will involve adjustments to the wording of several different questions in the survey – including a conjoint experiment and three A/B tests. Further details are in the attached statistical analysis plan.

Experimental design

The structure of the conjoint (discrete-choice) experiment follows the approach set out by Hainmueller, Hopkins & Yamamoto (2014).
Respondents will be presented with two scenarios for a programme evaluation, each of which consists of a random assortment of the pre-defined levels of each of the six attributes. They will be asked which of the two scenarios they prefer. Each respondent will be asked to make five such pairwise comparisons.

In the A/B tests, each respondent will be randomly assigned to one of two arms. The arms are set out in the attached analysis plan.

Design details

Randomization method
Randomization will be carried out within the Medallia survey interface, using simple randomization with no stratification. Randomization for each of the experiments will be carried out independently.
Randomization unit
Individual survey respondent
Clusters
400 individual survey respondents
Observations
400 individual survey respondents
Arms
Not applicable for the conjoint experiment. For the A/B tests, there are planned to be 200 respondents in each arm.
Primary outcomes
The conjoint experiment will test the extent to which policymakers prefer to carry out a randomized controlled trial rather than other types of evaluation, and how this is affected by the purpose of the evaluation, the timeframe in which results are needed, and the budget, set-up time and support available. Outcomes for the A/B tests are as follows: - Self-assessed level of understanding of policy experimentation - Level of own support for randomized experimentation - Expected level of support for randomized experimentation among those who are served or most closely affected by policies - Opinion about whether there should be more or less experimentation in R&I policy
Sponsors
Name: European Commission Location: Brussels Url: https://europa.eu/
Partners
Name: Technopolis Group Partner_type: private_company Url: https://technopolis-group.com/

Cited by

No citation data available for this registration yet.

Keywords