How Google Crawling and Indexing Works (in Plain English)

People often ask why a page is not on Google when, as far as they can tell, it should be. The answer almost always lies in one of three steps that happen before any ranking: crawling, indexing and serving.

Understanding these steps in plain terms takes the mystery out of SEO. Once you know how Google finds and stores pages, it becomes much easier to work out why one is missing and what to do about it.

Crawling: being found

Google discovers pages by following links and reading sitemaps. Its crawler visits a page, reads the content, and follows the links on it to find more pages. If nothing links to a page and it is not in your sitemap, Google may never find it.

Crawling can also be blocked accidentally — a robots file telling Google to stay away, or a page buried so deep nothing points to it. If Google cannot crawl a page, nothing else can happen.

Indexing: being stored

After crawling, Google decides whether to add the page to its index — the vast library it draws from to answer searches. Being crawled does not guarantee being indexed; Google may judge a page too thin, duplicated or low value to keep.

A no-index instruction, a canonical pointing elsewhere, or simply weak content can all stop a page being indexed. If a page is not indexed, it cannot appear in results no matter how relevant it is.

Serving: being ranked

Only indexed pages can be served in results, and then ranking decides the order. This is where relevance, quality, authority and the searcher’s intent come in. Most SEO advice focuses here, but the earlier two steps must succeed first.

When a page is not showing, work back through the steps: is it crawled, is it indexed, and only then is it ranking poorly? Google Search Console’s inspection tool tells you exactly where a given page stands.