How search engines work

Search engines consist of three main ingredients:

1. Crawling

2. Indexing

3. Picking the results

The process goes like this:

Crawling

Crawling or spidering means scanning the website, its sections, content, keywords,
headings, hyperlinks, images by thousands of small bots. Any data that can be found on
the website is crawled.
Crawlers detect all hypertext links on a website that point to other websites. Then
they parse those pages for new links over and over again. Bots crawl the whole internet
regularly to update the data.

Indexing

Once the website is crawled, the indexing takes place. Imagine the index as a gigantic
catalog or a library full of websites from all over the world. It usually takes some time
for a website to be indexed. From our experience, it’s from 1 to 10 days.

Pro tip: You can check what pages of your website have already been indexed by
using this search operator: site:domain.com

Furthermore, every time it’s changed, our good friend crawler scans it again. Keep in
mind that until the updates on the website are indexed, they won’t be visible in search
engines.

Picking the results

Results are critical for both developers and users. Once the internet user submits a
search query, the search engine digs into the index and pulls out matching results. It’s a
process of checking the query against billions of websites based on various
algorithms.

Companies running search engines (Google, Microsoft, Yahoo!) keep the exact
calculations of their algorithms in secret. Nonetheless, many ranking factors are well-
known.

Ranking factors
Most of these factors are proven, but some are just speculations or even myths. On top
of that, some are more important than others. Cyrus Sheppard from Zyppy made a nice
list of Google ranking factors.

You don’t have to know all of the ranking factors by heart to learn SEO, but it is good to
have at least a basic overview.
One of the most important factors, the backlink profile is based on the number and
quality of backlinks leading to a website. It’s a very simplified view on Google
approximation of the website’s authority. Each backlink is basically an analogy of an
academic citation.

Some other important ranking factors include (in no particular order):

 use of relevant keywords and phrases

 HTTPS

 link relevance

 grammar and spelling

 topical authority
  social sharing

 domain age

 AMP

 page layout

Ranking factors can be divided into on-page SEO factors (including technical SEO) and
link building or off-page SEO factors.

Quick tip: Check the domain authority of your or any other website to find its strengths
and weaknesses.

How people use search engines

The picture below represents one of the common user journeys in Google Search:
The interactions with search engines have evolved over the years. However, the
principle remains the same:

1. A need for a solution, information, or an answer

2. Typing the need in form of a query (keyword) into the search engine

3. Going through the first results

4. Clicking on one or more results (websites)

5. Scanning websites for the answer

6. Going through more results on the 1st SERP and/or changing the search query,
if the answer isn’t found.

Search engines market share

How do we classify search queries?

1. Navigational search queries

2. Informational search queries

3. Transactional search queries

Navigational search queries

They represent an intent to search for a particular brand or website. People tend to type
“youtube” or “google” into search engines rather than using browser’s history or
bookmarks.

Based on our case study where we analyzed 1,6 billion keywords, brands such as
YouTube, Facebook and Google reach the highest search volumes along with other
navigational search queries.

Informational search queries

These are submitted when users are searching for information. They aren’t looking for a
particular website, yet for an answer or guidance on how to do something. For example,
“How to bake pizza”.

Transactional search queries

This type is an intention to make a transaction. It usually comes with a product name
(Nike Airmax) or category (sneakers). Additionally, it can be written with “Where to buy
…”, “… price” or in a similar manner.

There are many blog posts on how to target a particular search query. However, it’s
may not be that easy in the future because of the increasing popularity of voice
assistants such as Siri, Google Now or Alexa.
Informational search queries can quickly transform to transactional by opening a new
app or giving an option to make a purchase.

SERP updates
Being on the first page of the organic search results is good, scoring the top three is
great but there’s only one winner, right? Or, is it? It’s a matter of perspective.

Websites all over the world are updated on a daily, weekly or monthly basis, …ok, some
are not but that’s a different case. The thing is that the internet grows every single day.
When new websites and changes are indexed by a search engine, the organic results
may change.

Another very important factor is Google algorithm which changes all the time. Minor
tweaks may not cause anything at all, but a major algorithm update can end up as an
earthquake.

Pro tip: Subscribe to our SERP volatility tracker to stay updated if there are any
algorithm changes.
What we’re trying to say is that even if you’re the winner, your positions can (and
probably will) be replaced by competitors the other day, and vice versa.

In the chart below, you can see the importance of the highest rankings in Google
depending on their organic click-through rate (CTR) distribution for May 2019
(based on data by Advanced Web Ranking).
SERP features
Of course, ranking first is important, but these days, you have to take into consideration
the so-called “zero position”.
Let’s take a look at the results for “How to bake potatoes” search query. The first result
is a Google featured snippet with all or the most important information, so you don’t
need to check the other results.
FURTHER READING
28 Google rich snippets you should know in 2020 [guide + infographic]

There are many SERP features (also called Rich snippets). Why should you care about
them? Rich snippets influence the behavior of users when they see the SERP. In
other words, the generic organic search results may have and in many cases, they do
have lower click-through rates (CTR).

It’s because the SERP features have bigger visual appeal and they often provide
enough information so the Google Search users don’t have to click on other results at
all or they click only on the featured results.

These are some of the most common rich snippets you’ll see in the SERP:

 Featured snippet

 Answer box

 Carousel (images, videos, products)

 Image pack

 Map pack

 Sponsored features (Google Ads, flights, shop on Google)

 Knowledge graph

 Top stories

 Events

 Sitelinks

The good thing is that there are ways to spot and analyze the impact of these enhanced
results. For example, SERPChecker will do the job.
Just type in the keyword, select the location and device type. This tool will show you
the search results and SERP features if there are any. It estimates the impact on a
scale from 0 to 5.

To find out more, click on the feature. You can also see the actual appearance of the
SERP if you click on the “Preview snapshot”.