AI Summaries and Online Search Behavior: Evidence from a Field Experiment on Google Search

How to cite

Sen, Ananya; Agarwal, Saharsh. 2026. "AI Summaries and Online Search Behavior: Evidence from a Field Experiment on Google Search." AEA RCT Registry. AEARCTR-0017393. https://doi.org/10.1257/rct.17393-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Ananya Sen* , Saharsh Agarwal

* Primary investigator

Registry ID
AEARCTR-0017393
DOI
https://doi.org/10.1257/rct.17393-1.0
First registered
Last updated
Status
In development
Trial dates
2026-01-07 – 2026-02-07
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/17393

Abstract

We study how AI-generated summaries on Google Search affect users’ engagement with both search results and downstream publisher content. To causally identify these effects, we run a large-scale field experiment using a custom-built browser extension that modifies what participants see on their search results page. The extension randomly assigns about 1,500 individuals across three conditions: a control group that sees the standard AI summaries, a “hide AI summaries” group, and an “AI mode” group that receives an enhanced AI-forward interface. Over a two-week period, we track on-page engagement by measuring link clicks, time spent on the search results page, and interactions with the AI elements. We also observe downstream behavior by recording activity on the websites users visit after clicking through. Our results provide new evidence on how AI summaries reshape online information consumption and the distribution of traffic across publishers.

Intervention

We randomly assign about 1,500 individuals across three conditions: (1) a control group that sees the standard AI summaries which is the status quo. (2) A “hide AI summaries” group in which our browser extension hides AI summaries for queries that are deemed eligible in Google search. (3) Finally, an “AI mode” group that receives an enhanced AI-forward interface whenever they search for a query on Google.

Experimental design

We randomly assign about 1,500 individuals across three conditions: (1) a control group that sees the standard AI summaries which is the status quo. (2) A “hide AI summaries” group in which our browser extension hides AI summaries for queries that are deemed eligible in Google search. (3) Finally, an “AI mode” group that receives an enhanced AI-forward interface whenever they search for a query on Google. Over two weeks, we track different engagement measures such as clicks on the search page, scroll depth, time spent on the search page, and downstream.

Apart from these outcome variables, we also want to focus on certain heterogeneity analyses. We will look at heterogeneity by query type, the position that the AI summary appears on, trust in AI-generated information online, and browsing behavior prior to the experiment. As additional analysis, we also want to look at how our treatment effects vary by age and gender.

Analysis across (1) and (2) will be done at the user-session level while comparisons with (3) will require more aggregation since the search experience would be different. In user-session analyses, standard errors will be clustered at the individual level.

Our sample criteria is the following:
(1) Google Chrome has to be the only browser used by the individual, (2) exclude if they use the device only for Prolific-like surveys. Using our extension, we also have criteria based on their browsing behavior: (a) Total number of days with any browsing history: minimum 10 out of 20 days. (b) Atleast 25 unique URL domains in last 20 days, (b) atleast 10 Google searches in last 20 days.

In our pilots, we have a handful of users who have an abnormally large number of searches on Google. Such users would be excluded from our analysis.

Design details

Randomization method
Randomization using computer software
Randomization unit
Individual
Clusters
Individual-level assignment
Observations
1500
Arms
1500
Primary outcomes
(1) Number or probability of clicks on the search page (across different types of clicks) (2) Time spent on the search page (3) Scroll depth on the search page. (4) Time spent on the clicked page
Secondary outcomes
Apart from the primary outcomes, we will also look at bounce rates which we can measure using our browser extension. We will also administer a post-experiment survey to elicit subjective attitudes about the overall search experience and the perceived value of AI summaries in online search.

Cited by

No citation data available for this registration yet.

Keywords