AI EVALUATIONS AND SCREENING - A Detailed Study on Human-AI Collaboration in Screening Efficiency and Decision-Making, 2024
How to cite
Ayoubi, Charles; Lane, Jacqueline; Boussioux, Leonard; Ho, Justin; Zhang, Miaomiao. 2024. "AI EVALUATIONS AND SCREENING - A Detailed Study on Human-AI Collaboration in Screening Efficiency and Decision-Making, 2024." AEA RCT Registry. AEARCTR-0013525. https://doi.org/10.1257/rct.13525-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
This study investigates the integration of artificial intelligence (AI) in the screening processes of early-stage innovations, traditionally conducted by human evaluators, across various professional and competitive settings. Through a randomized controlled trial involving around 400 participants from the MIT Solve expert internal screener team and from community leveraged startups screeners, this research explores whether AI-assisted human evaluators or AI-only evaluations enhance the efficiency and quality of decision-making compared to traditional human-only evaluations. Outcomes measured include the time efficiency of evaluations, consistency and convergence of decisions, evaluator confidence, and the overall quality of decisions across three conditions: control (no AI assistance), Treatment A (basic AI assistance), and Treatment B (advanced AI assistance providing detailed rationales). The findings aim to delineate the conditions under which human-AI collaboration optimizes evaluation outcomes of early-stage innovations, contributing to the broader discourse on effectively combining human intuition with AI’s processing capabilities. This could have significant implications for fields requiring precise and timely assessments, such as academic research, grant funding, and competitive selection processes, enhancing both theoretical understanding and practical applications of AI in evaluative tasks.
Intervention
This study employs three distinct intervention strategies to evaluate the impact of AI-assisted decision-making in the evaluation of early stage innovations:
Control Group: Participants will conduct evaluations without any technological assistance, mirroring traditional human-only evaluation processes.
Treatment A: This group involves the use of a generative AI tool that provides basic pass/fail recommendations to assist participants in their decision-making. This is intended to assess whether simple AI guidance can enhance the efficiency of evaluations compared to the traditional approach.
Treatment B: Participants in this group receive a more sophisticated form of AI assistance, which includes not only pass/fail recommendations but also detailed rationales behind each decision. This intervention aims to explore whether increased expalinability and depth in AI guidance can improve decision quality and evaluator confidence more significantly than basic AI assistance.
Experimental design
The experimental design for this study is structured as a three-arm randomized controlled trial (RCT) to evaluate the impact of AI-assisted decision-making in the screening of submissions for a global health equity challenge. The design is intended to test the efficiency and effectiveness of AI interventions compared to traditional human-only evaluation processes.
Control Group: Participants in this group perform evaluations manually, without any AI tools, serving as the baseline against which the AI-assisted groups are compared. This mimics the traditional process currently employed in most evaluative settings, allowing for a direct assessment of the impact of AI interventions.
Treatment A: In this arm, participants receive basic AI assistance, which provides binary (pass/fail) recommendations for each submission. This treatment tests the hypothesis that even minimal AI involvement can streamline the evaluation process and reduce the time and cognitive load on human evaluators.
Treatment B: Participants in this group use a more advanced AI tool that not only suggests binary outcomes but also provides detailed rationales for each recommendation. This treatment is designed to explore whether deeper AI integration, which includes providing context and explanations, can improve the quality of decisions and increase evaluator confidence and alignment with expert decisions.
The study's design allows for a controlled comparison across different levels of AI assistance, providing insights into how different AI models might enhance or interfere with human cognitive processes in evaluative tasks.
Design details
Cited by
No citation data available for this registration yet.