Do AI-Generated Assessments Affect Social Impact Scoring by Human Evaluators?

How to cite

Firpo, Teo; Ganguli, Ina; Banal-Estañol, Albert; Danilov, Anastasia. 2025. "Do AI-Generated Assessments Affect Social Impact Scoring by Human Evaluators?." AEA RCT Registry. AEARCTR-0017476. https://doi.org/10.1257/rct.17476-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Teo Firpo* , Ina Ganguli , Albert Banal-Estañol , Anastasia Danilov

* Primary investigator

Registry ID
AEARCTR-0017476
DOI
https://doi.org/10.1257/rct.17476-1.1
First registered
Last updated
Status
In development
Trial dates
2025-12-01 – 2026-02-28
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/17476

Abstract

This study examines whether exposure to AI-generated evaluative text influences how human assessors score the social impact of project proposals. Assessors are drawn from three groups with varying levels of expertise. In an online evaluation setting, assessors first review a proposal and provide an initial score focusing exclusively on its expected social impact. They are then shown AI-generated text assessing the social impact of the same proposal and are given the opportunity to revise their initial score and report their confidence in their decision. The study evaluates whether assessors update their social impact scores after seeing AI input and how confident they are in such updates.

Intervention

AI evaluations are randomized into three treatments:
1. Evidence-only
2. Argument-focused
3. Pure control

Experimental design

Assessors evaluate multiple project proposals in an online survey environment. For each proposal, assessors first provide an initial score assessing only the proposal’s expected social impact. Assessors are then shown AI-generated text evaluating the social impact of the same proposal and given the opportunity to revise their score and indicate their confidence. Assignment of AI-generated text is randomized at the proposal level. The experiment includes three groups of assessors with varying levels of expertise. Analysis compares updating behavior and confidence across assessors, proposals, and expertise groups.

Design details

Randomization method
Randomization of proposals, proposal order and treatments at the proposal-assessor level, conducted via Stata using the randtreat command and the date of randomization as seed.
Randomization unit
Proposal (same proposals randomize to different assessors in varying order and under different treatments). Each assessor sees proposals under different treatments. (Not all assessors see the pure control treatment, only a small subset).
Clusters
There are three groups of assessors: 1. External experts (EAs): the plan is recruit 30 experts. 2. Internal assessors with prior expertise in evaluating proposals under the same program (technical assessors, or TAs): 42 assessors. 3. Volunteer internal assessors who may or may not have evaluated proposals in the past but were not recently involved (volunteer assessors, VAs): 77 assessors.
Observations
EAs: 7 proposals each, for a total of 210 reviews TAs: different number by assessor, for a total of 292 reviews VAs: different number by assessor, for an estimated total of 300 reviews. All numbers are estimates since participants agreed to dedicate a specific amount of time to the task, without guarantees of a specific number of reviews each.
Arms
149 assessors across groups, each reviewing proposals under the different treatments (within subject design). An estimated 802 total reviews.
Primary outcomes
1. Binary update decision: whether the assessor updates their initial social impact score after exposure to AI-generated text (yes/no). 2. Confidence in update: self-reported confidence associated with the update decision. 3. Direction and magnitude of changes in social impact scores following AI exposure. 4. Differences in updating behavior and confidence across assessor expertise groups. 5. Predictive accuracy of scores (human and AI-generated) against real word outcomes
Secondary outcomes
Self assessment of accuracy of scoring with respect to real world outcomes. Assessment of AI accuracy of scoring with respect to real world outcomes.

Cited by

No citation data available for this registration yet.

Keywords