Technical foundations
Technical foundations: the files, headers and codes
This guide covers the mechanical layer of search: the HTTP responses, files and tags that tell a search engine what to fetch, what to index and which URL to keep. Each one controls exactly one thing, and most technical SEO problems come from expecting one to do another's job.
Almost every technical search problem is one of four things being used to do another’s job. Learning precisely what each control does — and, more usefully, what it does not — resolves most of them.
HTTP status codes
The status code is the first thing a crawler reads, and it determines everything
that follows. 200 means process this. 301 and 308 mean the resource moved
permanently and the target should inherit the original’s signals. 302 and
307 mean a temporary move, so the original URL stays canonical. 404 means
not found; 410 means deliberately gone, and is acted on faster. 429 and
5xx mean back off and try later — sustained server errors reduce crawl rate
across the whole site.
The worst response is a 200 that says “page not found” in its body. The
crawler treats it as real content, the indexer has to detect the lie, and crawl
budget is spent on nothing.
robots.txt
A plain text file at the root of a host, standardised as RFC 9309, listing which
paths each user-agent may fetch. It controls fetching only. A disallowed URL
can still be listed in results without a snippet if other pages link to it, and
a noindex directive on a disallowed page is never read.
XML sitemaps
A list of URLs you would like the engine to know about, with optional last modified dates. A sitemap is a discovery hint, not an instruction to crawl or index. Including a URL guarantees nothing; excluding one prevents nothing.
Canonical tags
rel=canonical tells the engine which URL of a duplicate set you consider
primary. It is a hint that is weighed against other signals — internal
links, redirects, sitemap entries, HTTPS — and the engine can and does choose
differently.
noindex
A meta tag or X-Robots-Tag header that removes a page from the index. It is
the only one of these controls that actually deindexes. The page must be
fetchable for it to work.
hreflang
Annotations declaring which language and region version of a page to serve. Return links must be reciprocal or the cluster is ignored. It affects which version is shown, not whether any version ranks.
Structured data
JSON-LD markup describing what a page is about, in schema.org vocabulary. It makes eligibility for certain result features possible; it is not a ranking signal in itself and it does not override what the page actually says.
Articles in this guide
- HTTP Status Codes That Actually Matter for Search
The status code is the first thing a crawler reads, and it decides everything after it. Here is what each code means to a search engine, and the traps.