Evaluating the efficacy and role of Artificial Intelligence (AI) in policy making

How to cite

Keenan, Michael; Karachiwalla, Naureen; Breisinger, Clemens; Koo, Jawoo; Mwangi, Christine; Kim, MinAh; Waruingi, Esther. 2023. "Evaluating the efficacy and role of Artificial Intelligence (AI) in policy making." AEA RCT Registry. AEARCTR-0012495. https://doi.org/10.1257/rct.12495-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Michael Keenan* , Naureen Karachiwalla , Clemens Breisinger , Jawoo Koo , Christine Mwangi , MinAh Kim , Esther Waruingi

* Primary investigator

Registry ID
AEARCTR-0012495
DOI
https://doi.org/10.1257/rct.12495-1.0
First registered
Last updated
Status
On going
Trial dates
2023-10-01 – 2023-12-01
Countries
Private
JEL codes
Z18; D04; O33; D83
Registry record
http://www.socialscienceregistry.org/trials/12495

Abstract

This study assesses the potential value of incorporating AI-powered chatbots like ChatGPT in the agrifood systems policymaking process. The study focuses on the International Food Policy Research Institute's (IFPRI) Country Strategy Support Programs (CSSPs) in the Global South across twelve study countries (Egypt, Ethiopia, Kenya, Rwanda, Ghana, Malawi, Nigeria, India, Sudan, Tajikistan, Uganda and Bangladesh). The study will assess the quality and perceptions of policy notes in the context of relevant issues for development, as determined by IFPRI’s participating country offices. The study seeks to answer three research questions; 1) For policymakers, policy analysts, and other stakeholders in low- and middle-income countries, how does the perceived quality of and intended engagement with policy notes differ between policy notes written entirely by researchers and policy notes written primarily by generative AI bots?; 2) To what extent does the disclosure of the use of AI in writing policy notes influence policymakers’, policy analysts’, and other stakeholders' perceived quality of and intended engagement with policy notes?; 3) What are policymakers', policy analysts', and other stakeholders' beliefs about others’ perceptions of quality and intended engagement with AI-generated and human-written policy notes?

Experimental design

The experimental design will follow a (semi-)factorial approach with the below treatment arms. Assignment to each treatment arm will be random. Given the nature of the study, balance across observables will need to be tested ex-post.
• T1: Respondents receive either an AI-generated policy note or a human-written policy note.
• T2: Respondents are told the policy-note is written by a human or is AI-generated.

Based on the above two treatments, respondents can be assigned to 1 of four treatment groups, denoted G1, G2, G3, and G4 below.

• G1: The respondents receive a policy note written by a human. They are told “This policy note was written by a team of local and international researchers.”
• G2: The respondents receive a policy note written by a human. They are told “During the preparation of this policy note, a team of local and international researchers used Artificial Intelligence to generate the content and structure of the note. The team has reviewed and edited the content as needed and takes full responsibility for the content of the policy note.”
• G3: The respondents receive a policy note written by an AI bot. They are told “This policy note was written by a team of local and international researchers.”
• G4: The respondents receive a policy note written by an AI bot. They are told “During the preparation of this policy note, a team of local and international researchers used Artificial Intelligence to generate the content and structure of the note. The team has reviewed and edited the content as needed and takes full responsibility for the content of the policy note.”

Follow-up questions are administered after the treatments are assigned.

Design details

Randomization method
The experiment uses systematic stratified random sampling procedure. Systematic randomization is needed because the study population is unknown a priori given that the population is determined by self-selection into the study. First, respondents are grouped into strata based on their country, position level (Entry level, Junior/associate, Mid-level, Senior-level (managers), or Executive or leadership) and gender (male or female). Once the stratum is assigned to a respondent, then respondents are randomized into one of the four treatment groups systematically. The first respondent in each strata is randomly assigned to one of four treatment groups, the second is randomly assigned to the remaining three treatment groups after removing the treatment group that the first is assigned to, the third is randomly assigned to the remaining two treatment groups, and the fourth is assigned to the last remaining treatment group. The process is repeated for all respondents based on their position (mod 4) in responding within their strata. The entire randomization process is completed within SurveyCTO, using computer-assisted randomization within the survey.
Randomization unit
Individual level, stratified randomization.
Clusters
0
Observations
300. However, the sample size may vary based on response rates.
Arms
75 per treatment arm. 300. However, the sample size may vary based on response rates.
Primary outcomes
- Measures policy note quality. - Measures of intended engagement with the policy note. - Measures of beliefs about peers’ assessment of quality and intended engagement.
Secondary outcomes
The amount of time spent reading the policy note will be a secondary outcome.

Cited by

No citation data available for this registration yet.

Keywords