Forward index

In plain English

When a search engine crawler downloads a webpage, it receives a raw stream of HTML markup, text blocks, image tags, and scripts. Before that information can be searched, the document must be parsed and structured.

A forward index is the first, natural way an information retrieval system stores this data: it maintains a direct list of documents, and for each document, it records every single word (token) that appears inside it. You can think of a forward index like a digital filing cabinet where each folder represents one specific document. Inside Document 1’s folder is a list of its words, their positions, and their frequency. While this document-to-words organization is excellent for examining what a specific document discusses, it is virtually unusable for answering search queries across billions of documents in real time.

An example

Suppose an experimental search crawler indexes three brief technical articles:

Document 1: "crawlers discover new web pages"
Document 2: "crawlers follow internal links"
Document 3: "indexing stores web pages"

The crawler builds a forward index by assigning each document a unique ID and mapping it to its parsed token sequence:

Forward Index:
Doc 1 -> ["crawlers", "discover", "new", "web", "pages"]
Doc 2 -> ["crawlers", "follow", "internal", "links"]
Doc 3 -> ["indexing", "stores", "web", "pages"]

Now, imagine a user searches for the query “crawlers.”

If the search engine relied exclusively on the forward index to answer the query, it would be forced to open Document 1 and read its word list, open Document 2 and read its word list, open Document 3 and read its list, and repeat that linear scan across every single document in the global web index. On an index containing tens of billions of web pages, a brute-force sequential search would take minutes per query, creating an unacceptably sluggish experience.

To make searches instant, the search engine takes the forward index and executes an inversion algorithm, flipping the data structure inside out into an inverted index:

Inverted Index:
"crawlers" -> [Doc 1, Doc 2]
"web"      -> [Doc 1, Doc 3]
"links"    -> [Doc 2]

With the inverted index constructed, looking up “crawlers” is an instantaneous dictionary lookup that returns Document 1 and Document 2 in a fraction of a millisecond.

Why it matters

The forward index is the crucial bridge connecting the web crawler to the inverted search index. Crawlers construct forward indexes as pages are downloaded, tokenized, and parsed. Without the forward index, search engines could not calculate document lengths, compute relative term frequencies (TF), filter stop words, or generate the posting lists that make modern internet search possible.

Related terms

Read the full guide to indexing, or explore our complete guide to forward index vs inverted index.