The Selective Tailwind effect of Artificial Intelligence

How to cite

Messinese, Danilo; Conti, Annamaria. 2025. "The Selective Tailwind effect of Artificial Intelligence." AEA RCT Registry. AEARCTR-0015201. https://doi.org/10.1257/rct.15201-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Danilo Messinese* , Annamaria Conti

* Primary investigator

Registry ID
AEARCTR-0015201
DOI
https://doi.org/10.1257/rct.15201-1.0
First registered
Last updated
Status
In development
Trial dates
2025-01-20 – 2025-03-15
Countries
Private
JEL codes
J24, L86, O31, O32, O33, D81, D83, M13
Registry record
http://www.socialscienceregistry.org/trials/15201

Abstract

What role does AI play in entrepreneurial decision making? We have partially explored this question by analyzing the impact of AI predictive tools on the performance of a large sample of US startups. We have exploited the random release of Google Analytics 4 (GA4) which introduced AI predictive tools especially useful for mobile app developers. Leveraging this shock in a difference-in-differences model, we found that post-GA4-release there is a boost in customer acquisition. However, the positive premium is driven by the upper tail of the treatment effect distribution, and not by marginal improvements. These effects are largest for innovative startups led by highly skilled founders. Shedding light on the mechanisms, we show that GA4 boosts the productivity of A/B testing tools. Overall, these findings suggest that AI predictive tools are useful for complementing skilled human capital in formulating new testable business hypotheses, especially relevant for the detection of breakthroughs.

To further explore our results obtained from large-scale data, we have built a platform for data analysis and plan to conduct a lab experiment on Prolific. The goal is to better understand the role of predictive AI tools for anomaly detection for the production of breakthrough innovations, as well as the interaction of these tools with entrepreneurs' human capital.

Intervention

The goal of this lab experiment is to test whether the use of AI for anomaly detection can improve the ability to identify innovation opportunities and whether these tools completment or not human capital. Our experiment encompasses two treatments. The first treatment is a nudge about the importance of anomalies detection to uncover novel ideas and business opportunities, delivered through a short video and an ad-hoc AI tool (details below). The second treatment is a nudge about how to leverage causal reasoning to produce innovations. This nudge will be delivered through a short video.

These two treatments will result in four groups of participants (2x2 matrix):
1. Group 1 will receive only the treatment about anomalies detection
2. Group 2 will receive only the treatment about causal reasoning
3. Group 3 will receive both treatments about anomalies detection & causal reasoning
4. Group 4 is the control group

Each group will access a platform developed by the research team to analyze data from a B2C software startup. This platform closely replicates Google Analytics 4 (GA4). Participants will be given a dataset simulating user behavior for a B2C food delivery app as of 2024, including variables such as user sessions, revenue, engagement, customer acquisition, and feature adoption. The dataset, in addition to data trends, contains anomalies that, if detected, could inspire innovative solutions. Notably, groups 1 and 3 will have access to a default option within the platform that highlights data anomalies as part of the treatment.

Experimental design

The goal of this study is to evaluate whether AI-based anomaly detection enhances the ability to identify innovation opportunities and to explore whether these tools complement or not human capital. Participants will be recruited via Prolific and tasked with analyzing data from a B2C startup, with the objective of proposing innovations based on their analysis.

Participants will be provided with an ad-hoc platform for data analysis developed by the research team with time-series about key performance metrics in 2024.

Design details

Randomization method
Randomization into the four groups described above will be performed by Qualtrics by randomly assigning participants to each of the four groups of the experiment.
Randomization unit
Individuals recruited on Prolific
Clusters
0
Observations
70 units per treatment group for a total of 280 units. We will target approximately 360 participants to include a buffer in case of attrition.
Arms
70 units per treatment group
Primary outcomes
- Number of innovations uncovered by participants - Level of innovativeness of innovations uncovered by participants
Secondary outcomes
Data-driven innovation

Cited by

No citation data available for this registration yet.

Keywords