What a search engine is
What a search engine is, and how the parts fit together
Before crawling, indexing or ranking makes sense individually, it helps to see the whole machine. This guide defines what a search engine is, separates it from the things it is regularly confused with, and shows how the four stages hand work to each other.
A search engine is a software system that discovers documents on a network, stores a processed representation of them, and returns an ordered list of those documents in response to a query. Three capabilities have to be present for something to be a search engine: it collects documents itself, it maintains its own index of them, and it ranks results rather than merely filtering them.
Four stages, one pipeline
The whole system is a pipeline, and each stage can only work on what the previous stage handed it.
Discovery and crawling finds URLs and fetches them over HTTP. A crawler starts from URLs it already knows, follows links, and adds new URLs to a queue. Nothing downstream can see a page the crawler never fetched.
Rendering and parsing turns raw bytes into structure. HTML is parsed into a document tree, JavaScript may be executed to produce the final content, links are extracted, and text is tokenised into individual terms.
Indexing writes those terms into a data structure built for lookup — an inverted index, which maps each term to the list of documents containing it, along with position and weighting information. Duplicate pages are collapsed to a canonical version at this stage.
Retrieval and ranking takes a query, finds candidate documents in the index, and scores them. Retrieval is fast and cheap; scoring is where relevance, quality, links, freshness and context are weighed together to produce an order.
What a search engine is not
A web browser requests and displays a document you already have the address for. It has no index and does not rank anything.
A web directory is a curated list of sites organised by humans into categories. Yahoo began this way. There is no crawler and no scoring.
A database query returns every row that matches a condition, in whatever order you specify. There is no relevance judgement — a row either matches or it does not.
An answer engine generates a response rather than returning a list of documents. Modern AI features sit on top of a search index rather than replacing it: the retrieval step is still a search engine doing its job.
Why the distinction matters in practice
Most confusion about “why is my page not ranking” resolves once you locate the problem in the pipeline. A page that was never crawled is a discovery problem. A page that was crawled but not indexed is a quality, duplication or directive problem. A page that is indexed but not ranking is a relevance and competition problem. These are three different failures with three different fixes, and the diagnostic tools for each are different.
Articles in this guide
The articles for this guide are being written. The guide above covers the whole topic in outline, and the article index lists everything published so far.