On this page
A crawler reads the status line before it reads a single byte of your content.
Everything downstream — rendering, indexing, ranking — happens only on the 200
branch. Getting the codes right is the cheapest technical work available.
The codes that matter
| Code | Meaning to a search engine | Typical use |
|---|---|---|
200 |
Real content. Process it. | Every page you want indexed |
301 |
Permanently moved. Transfer signals to the target. | Permanent URL changes, HTTPS and www consolidation |
308 |
Permanently moved, request method preserved. | Same as 301, safer for non-GET |
302 |
Temporarily moved. Keep the original canonical. | Maintenance pages, short-lived tests |
307 |
Temporarily moved, method preserved. | Same as 302 |
304 |
Not modified since last fetch. | Conditional requests, saves bandwidth |
404 |
Not found. Drop it eventually. | Pages that never existed or are gone |
410 |
Gone deliberately. Drop it sooner. | Content you have removed on purpose |
429 |
Too many requests. Slow down. | Rate limiting |
5xx |
Server broken. Retry later, crawl less. | Never intentional |
The traps
Redirect chains. Every hop is a separate fetch. A chain of four wastes crawl budget on three URLs that exist only to point elsewhere, and each hop is an opportunity to lose signal. Point the first URL at the final destination.
Redirecting everything to the homepage. When a removed page redirects
somewhere that does not answer what it answered, the engine treats it as a soft
404. Redirect to the closest genuine equivalent or return 404.
Soft 404s. A 200 with “page not found” in the body is worse than a real
404, because the crawler trusts the code and the indexer has to detect the
mismatch.
Sustained 5xx. Occasional server errors are handled gracefully. Prolonged
ones reduce crawl rate site-wide, and the recovery is slower than the decline.
Trusting a tool over the server. Browser extensions and third-party crawlers
cache. curl -I https://example.com/page/ is the ground truth, and it takes two
seconds.
Verifying in bulk
For a site of any size, check status codes as a set rather than one at a time.
Crawl your own site with a desktop crawler and export every non-200 response,
then compare that list against your XML sitemap. Every URL in a sitemap should
return 200. Anything else in there is a contradictory signal you are sending
on purpose.
To run a complete technical health audit across your headers, robots.txt, and rendering, use the 15-minute diagnostic checklist in Search Engine Basics.
Sources
Tier 1 is a search engine's own documentation or a primary standards document. Tier 2 is a reputable secondary publication or a peer-reviewed paper.
- How HTTP status codes, and network and DNS errors affect Google SearchGoogle Search CentralTier 1 source: primary documentation or a standards document
- HTTP Semantics (RFC 9110)IETFTier 1 source: primary documentation or a standards document
About the author
