Crawling
Crawling: how search engines find and fetch pages
Crawling is the stage where a search engine finds a URL and requests it over HTTP. Nothing else in search can happen until this succeeds. This guide covers how URLs are discovered, what a crawler sends and receives, what stops it, and how to prove from your own logs what was actually fetched.
A crawler is a program that requests URLs over HTTP and stores what comes back. Googlebot, Bingbot, Amazonbot and the rest are all variations on the same loop: take a URL off a queue, check whether the site’s rules allow fetching it, request it, record the response, extract any links, and add the new URLs to the queue. Everything that follows in search — rendering, indexing, ranking — operates on what this loop managed to collect.
Discovery comes before fetching
A crawler cannot request a URL it has never seen. URLs enter the queue in a small number of ways, and it is worth being precise about them because most “my page isn’t indexed” problems are really discovery problems:
- Links from pages the crawler has already fetched. This is the main source by a wide margin.
- XML sitemaps, which are a hint about which URLs exist, not an instruction to fetch them.
- Direct submission, such as the URL Inspection tool in Google Search Console or the IndexNow protocol used by Bing and Yandex.
- Redirect targets, canonical link targets, and URLs found in other structured references.
A page with no inbound links, not listed in a sitemap and never submitted is effectively invisible. It is not penalised, it is simply unknown. This is the single most common cause of a page never appearing in search, and it is why the homepage pipeline widget on this site deliberately includes one orphaned node.
Fetching is an HTTP conversation
When the crawler requests a URL it sends a normal HTTP request with a user-agent string identifying itself, and the server replies with a status code and, usually, a body. The status code decides what happens next:
200— the body is passed on to the next stage.301/308— the crawler follows the redirect and treats the target as the canonical location.302/307— the crawler follows it but keeps treating the original URL as the one that matters.404/410— nothing is passed on, and the URL is eventually dropped.429/5xx— the crawler backs off and retries later. Sustained server errors reduce how often the site is crawled at all.
Only the 200 path leads anywhere. A page that returns a 200 with an error
message in the body — a “soft 404” — is worse than a real 404, because the
crawler wastes budget on it and the indexing stage has to guess.
Rules that stop a crawl
Three separate mechanisms are routinely confused with each other:
robots.txtcontrols fetching. A disallowed URL is not requested. It can still be indexed from links alone, without a snippet, because the rule stops the fetch, not the listing.- A
noindexdirective controls indexing, and it can only work if the page is fetched. Blocking a URL inrobots.txtand addingnoindexto it is self-defeating: the crawler never sees the directive. - Authentication, IP blocks and rate limits stop the fetch at the server. From
the crawler’s side these look like
401,403or a timeout.
The technical foundations guide covers each of these files and headers in detail, and the articles below work through the specific failure modes.
Proving what happened
Crawling is one of the few parts of search you can observe directly rather than infer. Your server access logs record every request a crawler made, with the user-agent, the timestamp, the URL and the status code returned. Search Console’s Crawl Stats report shows the same picture from Google’s side. When the two disagree, the logs are the primary evidence.
Verify the user-agent before trusting it: anything can claim to be Googlebot, and a reverse DNS lookup on the requesting IP is the documented way to confirm a genuine Google crawler.
Articles in this guide
- What Is a Web Crawler and How Does It Work?
A web crawler is a program that fetches URLs over HTTP and follows the links it finds. Here is the fetch loop, the rules that stop it, and how to verify one.
- Why robots.txt Does Not Remove a Page From Google
Blocking a URL in robots.txt stops the fetch, not the listing. Here is why blocked pages still appear in results, and what actually removes them from the index.