Indexing
Indexing: how pages become searchable records
Indexing is the stage where a fetched page becomes a record that can be looked up in milliseconds. This guide covers rendering, tokenisation, the inverted index itself, canonicalisation, and the specific reasons a page that was crawled successfully still never gets indexed.
Crawling gives a search engine a pile of HTTP responses. Indexing turns that pile into something a query can be answered from in milliseconds. It is the least visible stage of search and the one where most “why isn’t my page showing up” problems actually live.
Rendering comes first
Raw HTML is parsed into a document object model. For pages that build their content with JavaScript, the engine must also execute that JavaScript to see the final text — a separate, more expensive step that happens on its own schedule. Google has described this as a deferred queue: the initial HTML is processed first, and rendering follows when resources allow.
The practical consequence is that content which only exists after JavaScript
runs is discovered later than content in the initial HTML, and it is not
discovered at all if the render fails, times out, or depends on resources
blocked in robots.txt.
Tokenisation and the inverted index
Once the text is available it is broken into tokens — roughly, words — and normalised. Case is folded, punctuation is stripped, and terms may be reduced to a common stem so that “crawling” and “crawler” can match a query for “crawl”.
Those tokens are written into an inverted index. Instead of storing “document 1 contains these words”, it stores “this word appears in these documents”, with the positions within each document. That inversion is what makes retrieval fast: answering a two-word query means intersecting two posting lists, not reading every document on the web.
Alongside the term data, the index stores per-document information the ranking stage will need: the canonical URL, the language, structured data, the title and description candidates, and quality signals accumulated over time.
Canonicalisation
The same content is often reachable at several URLs — with and without www,
with tracking parameters, over HTTP and HTTPS, or through a print view. The
engine groups these into a cluster and picks one URL as canonical. That choice
is informed by your rel=canonical hint, internal linking, redirects, sitemap
entries and HTTPS preference, but it is the engine’s decision, not yours.
When Search Console reports “Duplicate, Google chose a different canonical than user”, this is the stage that made that call.
Why a crawled page is not indexed
The common reasons, roughly in order of frequency:
- The page is a near-duplicate of another page and was clustered away.
- It carries a
noindexdirective in a meta tag orX-Robots-Tagheader. - It has thin or auto-generated content the engine judged not worth storing.
- It renders empty because the JavaScript failed or a required file was blocked.
- It is a soft 404 — a
200response whose body says the content is missing.
Indexing is selective by design. Being crawled earns a page consideration, not a place in the index.
Articles in this guide
- Crawled, Currently Not Indexed: What It Actually Means
Google fetched your page and chose not to store it. Here are the five real causes behind that Search Console status and how to tell which one applies.