Evaluating the efficacy and role of Artificial Intelligence (AI) in policy making
How to cite
Keenan, Michael; Karachiwalla, Naureen; Breisinger, Clemens; Koo, Jawoo; Mwangi, Christine; Kim, MinAh; Waruingi, Esther. 2023. "Evaluating the efficacy and role of Artificial Intelligence (AI) in policy making." AEA RCT Registry. AEARCTR-0012495. https://doi.org/10.1257/rct.12495-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This study assesses the potential value of incorporating AI-powered chatbots like ChatGPT in the agrifood systems policymaking process. The study focuses on the International Food Policy Research Institute's (IFPRI) Country Strategy Support Programs (CSSPs) in the Global South across twelve study countries (Egypt, Ethiopia, Kenya, Rwanda, Ghana, Malawi, Nigeria, India, Sudan, Tajikistan, Uganda and Bangladesh). The study will assess the quality and perceptions of policy notes in the context of relevant issues for development, as determined by IFPRI’s participating country offices. The study seeks to answer three research questions; 1) For policymakers, policy analysts, and other stakeholders in low- and middle-income countries, how does the perceived quality of and intended engagement with policy notes differ between policy notes written entirely by researchers and policy notes written primarily by generative AI bots?; 2) To what extent does the disclosure of the use of AI in writing policy notes influence policymakers’, policy analysts’, and other stakeholders' perceived quality of and intended engagement with policy notes?; 3) What are policymakers', policy analysts', and other stakeholders' beliefs about others’ perceptions of quality and intended engagement with AI-generated and human-written policy notes?
Experimental design
The experimental design will follow a (semi-)factorial approach with the below treatment arms. Assignment to each treatment arm will be random. Given the nature of the study, balance across observables will need to be tested ex-post.
• T1: Respondents receive either an AI-generated policy note or a human-written policy note.
• T2: Respondents are told the policy-note is written by a human or is AI-generated.
Based on the above two treatments, respondents can be assigned to 1 of four treatment groups, denoted G1, G2, G3, and G4 below.
• G1: The respondents receive a policy note written by a human. They are told “This policy note was written by a team of local and international researchers.”
• G2: The respondents receive a policy note written by a human. They are told “During the preparation of this policy note, a team of local and international researchers used Artificial Intelligence to generate the content and structure of the note. The team has reviewed and edited the content as needed and takes full responsibility for the content of the policy note.”
• G3: The respondents receive a policy note written by an AI bot. They are told “This policy note was written by a team of local and international researchers.”
• G4: The respondents receive a policy note written by an AI bot. They are told “During the preparation of this policy note, a team of local and international researchers used Artificial Intelligence to generate the content and structure of the note. The team has reviewed and edited the content as needed and takes full responsibility for the content of the policy note.”
Follow-up questions are administered after the treatments are assigned.
Design details
Cited by
No citation data available for this registration yet.