Effect of A.I. Evaluators on Creative Output

How to cite

Sandvik, Jason. 2024. "Effect of A.I. Evaluators on Creative Output." AEA RCT Registry. AEARCTR-0014860. https://doi.org/10.1257/rct.14860-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Jason Sandvik*

* Primary investigator

Registry ID
AEARCTR-0014860
DOI
https://doi.org/10.1257/rct.14860-1.1
First registered
Last updated
Status
Completed
Trial dates
2024-11-18 – 2024-12-02
Countries
United States of America ()
Registry record
http://www.socialscienceregistry.org/trials/14860

Abstract

Previous research suggests that quantity-based incentives can be just as effective as quality-based incentives in driving creative outcomes. A potentially non-trivial cost to quantity-based incentives, though, is that it increases the number of items that managers need to evaluate. Artificial intelligence (A.I.) programs provide a way for managers to evaluate creative content quickly, allowing them to assess the quality of a large amount of content in a relatively short amount of time. However, the use of A.I. to evaluate the quality of creative content could discourage workers from developing and sharing their innovative ideas if they are uncertain as to how A.I. will evaluate their content, or if they are meaningfully averse towards algorithmic evaluations of their creative content. We conduct an experiment to see how participation in a creative idea-sharing contest is impacted based on the types of evaluators that are used to evaluate the submissions—A.I. evaluators versus human evaluators—and based on whether quantity-based incentives or quality-based incentives are used.

Intervention

Participants will begin by answering basic demographic questions in a Qualtrics survey. They will then answer questions about their feelings towards human evaluators and A.I. evaluators, and they'll provide details about their experiences in and knowledge of New York City.

Participants will be asked whether or not they want to create and submit short promotions ("promos", of 300 characters or less) to get people to visit New York City. They will either be told that their promos will be evaluated by "several people with marketing experience" or "an artificial intelligence (A.I.) program that is specifically designed for use in marketing." Each promo will be evaluated on a 0-100 scale based on how likely it is that someone will click on the promo to learn more. To incentivize participation, they will either be told they will "receive $0.50 (50 cents) for each promo they submit, so long as the promo is evaluated with a score above 33. If it is evaluated with a score of 33 or lower, no payment will be provided for that particular promo" or "receive $50 for each promo you submit that is among the 10 highest-scoring promos in the entire contest. Approximately one out of every ten participants is expected to win $50." In both cases, they will be told that they can submit up to 10 different promos to increase their potential payout.

After learning all this information, they'll be asked if they want to enter the contest. For those that say no, the survey will end and they'll receive the pre-contracted payment amount. For those that say yes, they'll have approximately one hour to submit their promos. Once they finish, they'll receive the pre-contracted payment amount and then, after the promos are evaluated, they'll receive additional payments given the parameters of their contest and the quality of their submitted promos.

Experimental design

Participants will begin by answering basic demographic questions in a Qualtrics survey. They will then answer questions about their feelings towards human evaluators and A.I. evaluators, and they'll provide details about their experiences in and knowledge of New York City.

Participants will be asked whether or not they want to create and submit short promotions ("promos", of 300 characters or less) to get people to visit New York City. They will either be told that their promos will be evaluated by "several people with marketing experience" (human evaluator) or "an artificial intelligence (A.I.) program that is specifically designed for use in marketing" (A.I. evaluator). Each promo will be evaluated on a 0-100 scale based on how likely it is that someone will click on the promo to learn more. To incentivize participation, they will either be told they will "receive $0.50 (50 cents) for each promo they submit, so long as the promo is evaluated with a score above 33. If it is evaluated with a score of 33 or lower, no payment will be provided for that particular promo" (quantity-based incentives) or "receive $50 for each promo you submit that is among the 10 highest-scoring promos in the entire contest. Approximately one out of every ten participants is expected to win $50" (quality-based incentives). In both cases, they will be told that they can submit up to 10 different promos to increase their potential payout.

After learning all this information, they'll be asked if they want to enter the contest. For those that say no, the survey will end and they'll receive the pre-contracted payment amount. For those that say yes, they'll have approximately one hour to submit their promos. Once they finish, they'll receive the pre-contracted payment amount and then, after the promos are evaluated, they'll receive additional payments given the parameters of their contest and the quality of their submitted promos.

Randomization will occur via the block-randomization feature in Qualtrics. 25% of participants will be randomized into the A.I. evaluator and quantity-based incentives treatment; 25% of participants will be randomized into the A.I. evaluator and quality-based incentives treatment; 25% of participants will be randomized into the human evaluator and quantity-based incentives treatment; and 25% of participants will be randomized into the human evaluator and quality-based incentives treatment.

The only differences in content across the four treatment cells will be the language about the types of evaluators who will evaluate the promos and the types of incentives in place for participating in the contest.

Design details

Randomization method
Randomization via the Qualtrics block-randomization feature.
Randomization unit
Individual labor market participant
Clusters
1,000 labor market participants
Observations
1,000 labor market participants
Arms
250 labor market participants in A.I. evaluator x quantity-based incentives 250 labor market participants in A.I. evaluator x quality-based incentives 250 labor market participants in human evaluator x quantity-based incentives 250 labor market participants in human evaluator x quality-based incentives
Primary outcomes
The decision to enter the contest (yes/no response to the question asking if they want to participate). The number of promos submitted in the contest for those who enter.
Secondary outcomes
Average quality of each promo, as evaluated on the 0-100 scale by (i) the A.I evaluator, (ii) the human evaluators, and (iii) all evaluators. We will also consider how participation (primary outcomes) and promo quality (secondary outcomes) vary based on the demographic characteristics of the participants (e.g., whether they from a demographic group that is traditionally underrepresented in creative endeavors or whether they exhibit ex ante high levels of algorithm aversion).

Cited by

No citation data available for this registration yet.

Keywords