How Google Search Works: Crawling to Ranking Explained

Ever wondered how google searches anything from a database of more than 100 million gigabytes that too in few milliseconds !!
Hey all, Welcome to my Second blog on how google performs your search query in the backend .We will learn the entire process in simple language.

😃

No prerequisites required !!

Implementation Overview

Googles search architecture is divided into 3 phases namely crawling, indexing and searching. Lets assume you are owner of an e-commerce business like amazon and flipkart and you have just launched your website. Now, you want Google to include your website in search results when people look for products like watches or clothes — and improve your SEO (SEO = Search Engine Optimization = Ranking your website higher on search engines)

Architecture

Crawling 🕷️:

Google’s bots(known as crawlers) goes through(scans) a website and store all the data it can access on the website in its local servers. This process is called crawling. These local servers are also known as crawl servers .
Every website in this world is crawled by google’s bots.
There are two main ways bots come to your website
First is Recursive Crawling: As the name suggests, it’s recursive — when bots are crawling another website and that site has a link to your website, the bots will follow the link and crawl your site.
Another is Manual way : You invite the bots to crawl your website by filling up a form on Google Search Console and then within 48 hrs max your website will be crawled.
Only websites that are live/deployed on internet can be crawled.
You can restrict which parts of your website bots are allowed to crawl, especially for private or sensitive sections.

💡

For your website to appear in Google's search results, it must first be crawled by Google's bots.

Also, whenever you update your website (add new products, change content), Google needs to re-crawl it so the latest version appears in search results.
There are 2 ways to trigger re-crawling:
Google Schedules Recrawls Automatically (like a regular check-up) for all sites which it has crawled in the past. Popular websites automatically get more frequently crawled.
Another is again the Manual way.
There are other methods too like Sitemaps.
In the 1990s, as the amount of data on the internet grew rapidly, Google developed web crawling out of necessity. Initially manual crawling of famous websites like university websites which contain a lot of information about everything, were done by developers, leading to the beginning of the global spread of web crawlers.

Indexing 📋

Now all that huge data collected by bots is analysed, summarised and important features needed in search process are extracted and stored. This process is called as Indexing and is done by Indexers.
Now this categorised and short data is stored in Googles data centers which are distributed across world under Search Index data field (Googles data centers have other type of data sections also for eg the gmail data, google photos, google drive data etc).
Images, videos, documents and other heavy data is not stored!
Indexer creates a map like structure called Inverted index for eg —

    "shoes" → [Page_12, Page_45, Page_98]
    "skirts" → [Page_5, Page_55]
    "best" → [Page_12, Page_88]

Each page contains data like —

    Page_45 = {
      URL: "example.com/shoes",
      Title: "Best shoes under Rs1500",
      Snippet: "Affordable, durable and pocket friendly products...",
      DateIndexed: 2025-05-01
       }

This structure allows fast lookups — we’ll see how next.

Searching 🔍

Whenever user types search query like “Best shoes which are under Rs1700” then Tokenization process occurs which is converting each word to its root/basic form and eliminating common/stop words like “which, are”.
Now these words are searched in the Inverted Index (using highly efficient data structures and algorithms) and the filtered pages are ranked on basis of a lot of factors like content quality, speed, backlinks, freshness. This is decided by Ranking Algorithms.
Finally, this ranked result is returned to the user — all within milliseconds! ⚡

If this article was worth your time pls like and subscribe to my newsletter.

Thank you ! 💫

Decoding Google Search: The Silent Magic!