In This Article:

Project Overview

In the Search Meta-Analysis Project, we analyzed 471 different search queries from four usability-testing studies conducted between 2016 and 2018. Two of those studies used eyetracking equipment to follow participants’ eye movements. The studies were conducted in Raleigh, Kansas City, and San Francisco. 83 people participated in the four studies. They represented a diverse set of backgrounds, experience levels, and interests. All participants were over the age of 18.

The studies we analyzed tested a broad variety of tasks, and users visited a broad variety of websites while performing them. These studies didn’t explicitly test search — meaning that we did not ask users to use (or not use) search. The tasks were general (e.g., What’s the tallest building in the world? and Your friend is planning to refinish his clawfoot bathtub, find some instructions to help him.).  Many were real tasks that participants brought to the study. For example, one participant was planning a birthday party, so she searched for event spaces and catering companies to help her do that.

Today’s users frequently turn to search engines as a step in performing general tasks, so we observed a lot of search engine use in the process of those tasks. This search-engine use is what we analyzed in the Search Meta-Analysis Project. Search use as a side effect, in the context of a broad task, is likely to be more realistic than search use in a study specifically targeting search.

The search-engine-use segments of the videos from the four studies were coded for metrics including:

  • Query length
  • Which elements on the search-engine results page (SERP) were selected
  • Which elements participants looked at
  • How far down the SERP each participant looked
  • The number of times the participant reformulated her query

Articles & Videos

The findings from this project are explained in the following articles and videos:

Limitations of the Project

Recent Changes in Google SERPs

Users were not told what search engine to use (or, indeed, whether to use a search engine at all to achieve their tasks). But since these studies were run in the United States where Google has an overwhelming market share, the vast majority of the people used Google for their searches. Some participants used Bing (because it was the default page opened by an eyetracker in one of the studies) and a few used DuckDuckGo.

Google constantly changes its search features and page layouts. Since these studies were conducted, Google has made the following notable changes:

  • Tabs (Images, Videos, News) now have icons.
  • Google’s search bar stays fixed to the top of the page when the user scrolls on the SERP.
  • Additional personalization features, such as Recent Activity, were introduced.

These changes — as well as future changes to come — may influence user behavior. However, we believe that the majority of our findings are accurate today, with a few exceptions. Those exceptions are noted in the analysis articles.

Difficulties in Quantifying SERPs

While performing this analysis, we encountered several challenges as a result of the complexity of modern search-engine results pages. These included:

  • Different queries produce radically different SERP layouts and features.
  • The number of ads may change if a participant returns to the SERP from another page.
  • The size and position of different SERP features (e.g., knowledge panels) varies.
  • Results chosen for SERP features are constantly changing.
  • Ads and SERP features have different visual treatments on different search engines.
  • Sometimes ads contain images.

To address this complexity and enable our data collection, we created a flexible schema to define the various components of SERPs.