Limits and capabilities of LLM

How to cite

Hakimov, Rustamdjan; Bohren, Noah; Lalive, Rafael. 2023. "Limits and capabilities of LLM." AEA RCT Registry. AEARCTR-0011584. https://doi.org/10.1257/rct.11584-1.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Rustamdjan Hakimov* , Noah Bohren , Rafael Lalive

* Primary investigator

Registry ID
AEARCTR-0011584
DOI
https://doi.org/10.1257/rct.11584-1.1
First registered
Last updated
Status
In development
Trial dates
2023-06-18 – 2023-09-30
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/11584

Abstract

We are conducting an experiment to determine whether state-of-the-art language prediction models can outperform humans in areas of creativity and strategic intelligence. Comparing human performance to these advanced models is of paramount importance. This comparison sheds light on the level of competition that language models can present and the range of their potential applications, especially in areas traditionally seen as beyond their capabilities. Additionally, we also study effect of competition with artificial intelligence on creativity of human subjects, both for those who perform the tasks and for the evaluators, who might alter their perceptions of creativity when informed that some of the responses were generated by AI.

Intervention

In the first experiment, the interventions vary the information about competitors.

In the second experiment, participants rate texts, and interventions vary information about those who submitted the text for ratings.

Experimental design

In the first experiment, Prolific participants undertake creative and strategic tasks. The interventions vary the information about competitors.

In the second experiment, Prolific participants rate creative responses to one of two questions.The interventions vary information about those who submitted the text for ratings.

Design details

Randomization method
based on randomization in qualtrics
Randomization unit
participant
Clusters
1000 for experiment 1+300 for new treatment 4200 for experiment 2
Observations
1000 for experiment 1+300 for new treatment 4200 for experiment 2
Arms
1000 for experiment 1+300 for new treatment 4200 for experiment 2
Primary outcomes
We will estimate the impact of the treatment on various outcomes: (1) creativity measures (overall, uniqueness, surprise, and value) by prolific workers; (2) creativity measures (overall, uniqueness, surprise, and value) by RAs and creative workers; (3) the number of points won in strategic tasks in the last 12 rounds; and (4) the number of "rock" moves in the last 12 rounds.

Cited by

No citation data available for this registration yet.

Keywords