robots.txt: The Complete Guide With Examples

The robots.txt file is a plain text document placed at the root of a web server that instructs automated web crawlers which URL paths they are permitted to request. Formalized under RFC 9309 as the Robots Exclusion Protocol, it governs crawler fetching behavior rather than search engine indexation. Compliant search bots check this file before requesting pages across your domain.

What robots.txt controls, and what it does not

The single most consequential misunderstanding in search engine optimization is confusing crawl prevention with index removal. A robots.txt file controls fetching permissions over HTTP; it instructs compliant bots which paths they may download over the network. It does not control indexing, rendering directives, or link graph calculations.

If a page is disallowed in robots.txt, Googlebot will not fetch the HTML file, execute its scripts, or inspect its metadata. However, if external websites link to that disallowed address, Google already knows the URL exists. Because Google cannot fetch the document to read on-page tags, it frequently indexes the bare URL anyway, displaying an empty snippet. We explore this exact phenomenon in our dedicated technical guide explaining why robots.txt does not deindex pages.

Furthermore, search engines completely ignore unauthorized custom directives inside robots.txt. Some developers mistakenly add lines like noindex: /private/ to their file, assuming it suppresses search results. Google explicitly terminated support for experimental noindex directives in robots.txt in 2019, and the official RFC 9309 specification does not support it. To remove a page from search results, you must allow the crawler to fetch the document so it can read a noindex tag.

Finally, compliance with robots.txt is entirely voluntary on the client side. Well-behaved commercial crawlers like Googlebot and Bingbot honor your rules strictly. Malicious scrapers, email harvesters, and vulnerability scanners routinely ignore robots.txt instructions. You should never rely on robots.txt as an access control mechanism or security boundary for confidential data.

Where the file lives and how crawlers find it

A robots.txt file must reside in the top-level root directory of the host server using lowercase characters. When a crawler visits any URL on your website, its first action is to request https://example.com/robots.txt. If you place the file inside a subdirectory like /assets/robots.txt or use uppercase letters like /Robots.txt, crawlers will fail to locate it.

Valid Path:   https://example.com/robots.txt
Invalid Path: https://example.com/assets/robots.txt
Invalid Path: https://example.com/Robots.TXT
Invalid Path: https://example.com/subfolder/robots.txt

Robots policies apply strictly on a per-protocol, per-host, and per-port basis. A robots.txt file served over HTTP does not govern HTTPS requests. Similarly, a file placed on the apex domain example.com has zero authority over subdomains like blog.example.com or shop.example.com. Each distinct subdomain must serve its own independent robots.txt file at its own root.

Web servers must serve robots.txt as plain text using the MIME type text/plain encoded in UTF-8. Serving the file with an HTML content type or encoding it with a Byte Order Mark can cause parsing engines to misinterpret or reject your rules. Search engines cache the parsed contents of your robots.txt file for up to twenty-four hours to minimize redundant server requests.

The syntax, line by line

The syntax of a robots.txt file consists of line-oriented directives grouped into distinct rule blocks. Empty lines separate rule records, while comment lines begin with a hash character and are ignored by parsers. The four standard directives recognized across search engines are User-agent, Disallow, Allow, and Sitemap.

The User-agent directive specifies which automated crawler the subsequent rules apply to. You can target specific bots by name, such as Googlebot or Bingbot, or use an asterisk wildcard * to declare fallback rules for all compliant crawlers. Every rule block must begin with at least one User-agent declaration.

The Disallow directive specifies a URL path prefix that matching crawlers are forbidden from requesting. If the path value is left empty, the directive acts as a total permission, indicating that no paths are disallowed. If the path value is a forward slash /, the crawler is forbidden from requesting any URL on the entire host.

The Allow directive explicitly permits crawlers to request a specific path or file located within an otherwise disallowed directory. It acts as an exception mechanism, allowing engineers to block an entire directory while keeping specific assets accessible.

The Sitemap directive provides crawlers with the absolute URL of an XML sitemap. Unlike crawl rules, the Sitemap directive is independent of user-agent blocks and applies globally across all search engines. You can declare multiple sitemap lines anywhere in the file.

# Global rules for all compliant bots
User-agent: *
Disallow: /admin/
Allow: /admin/login.php

# Reference to the XML sitemap index
Sitemap: https://example.com/sitemap_index.xml

Path matching and wildcards

Path matching in RFC 9309 operates via prefix matching from the beginning of the URL path. When you write Disallow: /shop, the rule matches /shop, /shop/, /shopping, and /shop-now.html. If you intend to restrict crawling exclusively to the directory and its children, you must append a trailing forward slash: Disallow: /shop/.

The asterisk * functions as a wildcard representing zero or more occurrences of any valid character. This allows you to match file extensions or dynamic query parameters across your entire site architecture. For instance, Disallow: /*.pdf$ blocks every PDF document regardless of which subdirectory houses it.

The dollar sign $ designates the exact end of a URL path. Because robots.txt uses prefix matching by default, writing Disallow: /exact matches /exact-match and /exact/page. Writing Disallow: /exact$ restricts the match strictly to that exact string ending.

URL paths in robots.txt are strictly case-sensitive. The directive Disallow: /archive/ will not prevent a crawler from requesting /Archive/ or /ARCHIVE/. If your web server serves URLs with uppercase letters or mixed casing, your path definitions must mirror those exact string values.

Rule precedence when rules conflict

When multiple rules match a single URL path, compliant crawlers resolve conflicts using the deterministic precedence rules codified in RFC 9309. The most fundamental principle of modern robots.txt evaluation is that the most specific rule wins. Specificity is determined by comparing the character length of the matching path expressions.

If an Allow directive and a Disallow directive both match a given URL, the rule with the greater number of characters takes precedence. For example, if you declare Disallow: /catalog/ (9 characters) and Allow: /catalog/products/ (18 characters), the URL /catalog/products/shoes.html is permitted because the Allow rule has a longer matching character length.

In the event that an Allow and a Disallow rule match the exact same character length on a URL, RFC 9309 specifies that Allow wins. The table below illustrates how different rule combinations resolve across sample URLs:

Historically, search engines differed slightly in how they broke ties prior to RFC 9309 standardization. Today, major search engines including Google and Bing adhere strictly to the longest-match-wins standard, providing predictable behavior across all compliant crawlers.

Grouping rules by user-agent

Crawlers evaluate robots.txt files by searching for the single user-agent block that best matches their identity. An automated crawler will never merge, combine, or cascade directives from multiple user-agent groups. Once a crawler finds a specific group matching its name, it obeys that group exclusively and ignores everything else.

Consider a robots.txt file that defines rules for User-agent: * and separate rules for User-agent: Googlebot. When Googlebot inspects this file, it locates the User-agent: Googlebot record. It reads the directives inside that block and completely discards all directives written under User-agent: *.

User-agent: *
Disallow: /private/
Disallow: /reports/

User-agent: Googlebot
Disallow: /staging/

In the configuration above, Googlebot is allowed to crawl /private/ and /reports/ because those restrictions only exist in the wildcard group. If you want a specific bot to inherit general site restrictions, you must explicitly duplicate those disallow lines within the bot-specific block.

Crawlers also handle multiple user-agent lines grouped together at the top of a single block. If you write User-agent: Googlebot immediately followed by User-agent: Bingbot, both crawlers will execute the directives contained in that shared group.

Ten complete robots.txt files you can copy

Real-world web architectures require tailored robots configurations that reflect specific technical stacks, privacy requirements, and server environments. Below is a reference library of ten complete, production-tested robots.txt files designed for distinct publishing scenarios.

1. Standard Production Website
2. Complete Staging and Development Environment
3. Blocking Internal Search Results Pages
4. Blocking Tracking Parameters while Allowing Clean URLs
5. Allowing a Single File Inside a Disallowed Directory
6. Blocking Known AI Training Scrapers
7. Allowing AI Search Bots while Blocking Training Scrapers
8. WordPress Production Environment
9. E-Commerce Faceted Navigation and Filter Parameters
10. Development Subdomain with Complete Lockdown

Example 1: Standard Production Website

This baseline configuration is ideal for blogs, corporate portals, and marketing sites that want full indexation while protecting sensitive admin areas and pointing crawlers to sitemaps. It ensures standard content remains accessible while restricting checkout and management routes.

User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /checkout/

Sitemap: https://example.com/sitemap_index.xml

Line 1 declares that the following rules apply to all compliant web crawlers. Line 2 forbids crawlers from fetching any URL beginning with /admin/, protecting management interfaces. Lines 3 and 4 block user authentication paths under /login/ and dynamic checkout funnels. Line 6 provides an absolute link to the primary XML sitemap.

Example 2: Complete Staging and Development Environment

This configuration completely shuts down all crawling activity across an entire web domain, preventing staging environments from consuming crawl resources. It should be applied to development instances, QA environments, and internal preview servers.

User-agent: *
Disallow: /

Line 1 applies the rule to all compliant automated bots. Line 2 uses a single forward slash to disallow every single URL path on the host, preventing search engines from requesting any files on the staging server.

Example 3: Blocking Internal Search Results Pages

Search engines strongly discourage the indexing of internal search query results because they create infinite low-quality pages that waste crawl resources. Blocking these patterns prevents crawlers from entering spider traps created by user search terms.

User-agent: *
Disallow: /search/
Disallow: /search?*
Disallow: /*?s=*

Sitemap: https://example.com/sitemap.xml

Line 1 targets all compliant web crawlers. Lines 2 and 3 block path-based search queries like /search/keyword and query-string searches beginning with /search?. Line 4 uses wildcards to block traditional search parameters like ?s= anywhere on the domain.

Example 4: Blocking Tracking Parameters while Allowing Clean URLs

Marketing campaigns often append tracking parameters that cause severe duplicate crawling loops across otherwise identical canonical documents. Restricting these parameter patterns preserves crawl budget for unique content.

User-agent: *
Disallow: /*?*utm_
Disallow: /*?*fbclid
Disallow: /*?*gclid

Sitemap: https://example.com/sitemap.xml

Line 1 addresses all web crawlers. Lines 2 and 3 use wildcards to disallow any URL containing the Google Analytics tracking string utm_ or Facebook click tracking strings. Line 4 blocks Google Ads click identifier parameters while allowing base pages to be crawled cleanly.

Example 5: Allowing a Single File Inside a Disallowed Directory

This configuration demonstrates the longest-match precedence rule by blocking an entire assets folder while making a single verification file accessible. It allows public access to one specific whitepaper while protecting the remainder of the directory.

User-agent: *
Disallow: /downloads/
Allow: /downloads/public-whitepaper.pdf

Sitemap: https://example.com/sitemap.xml

Line 1 targets all web crawlers, while Line 2 disallows crawling across all files inside the /downloads/ directory. Line 3 defines an explicit Allow rule for a specific PDF file. Because Line 3 has a longer character length, crawlers are permitted to fetch the whitepaper while ignoring all other downloads.

Example 6: Blocking Known AI Training Scrapers

Website owners who wish to prevent automated commercial scrapers from consuming content for machine learning training can target specific AI crawler user-agents. This setup stops training scrapers without interfering with conventional organic search crawlers.

User-agent: GPTBot
User-agent: CCBot
User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /private/
Sitemap: https://example.com/sitemap.xml

Lines 1 through 3 group the primary user-agents for OpenAI, Common Crawl, and Anthropic. Line 4 disallows these AI scrapers from requesting any URLs across the domain. Lines 6 through 8 maintain standard crawling permissions for search engines like Google and Bing.

Example 7: Allowing AI Search Bots while Blocking Training Scrapers

Certain publishers wish to appear in conversational AI search engines like ChatGPT Search while disallowing bulk offline model training datasets. This file blocks model training extraction while keeping the site crawlable for general search.

User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /internal/
Sitemap: https://example.com/sitemap.xml

Line 1 targets Google-Extended, which governs Gemini training data extraction, and Line 2 targets Common Crawl. Line 3 blocks both training scrapers from all paths. The wildcard group maintains normal crawling access for standard search bots.

Example 8: WordPress Production Environment

The standard configuration for WordPress sites blocks backend administrative scripts while keeping Ajax endpoints accessible for client-side rendering. This prevents Googlebot from getting trapped in admin screens while ensuring themes render correctly.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Line 1 applies to all search crawlers. Line 2 disallows the entire WordPress management directory, while Line 3 creates an explicit exception for admin-ajax.php. Line 5 links to the native WordPress core XML sitemap.

Example 9: E-Commerce Faceted Navigation and Filter Parameters

E-commerce websites frequently suffer from crawl bloat when multi-select category filters create millions of unindexable URL permutations. Blocking faceted parameters keeps search bots focused on primary category and product URLs.

User-agent: *
Disallow: /*?*sort=
Disallow: /*?*price=
Disallow: /*?*color=
Disallow: /*?*filter=

Sitemap: https://example.com/sitemap_index.xml

Line 1 targets all web crawlers. Lines 2 through 5 block sorting, price range, color variation, and general faceted navigation parameters. This ensures bots focus their crawl capacity exclusively on primary canonical pages.

Example 10: Development Subdomain with Complete Lockdown

Development and pre-production servers should completely block crawlers while providing clear documentation for engineering teams. Placing this file on dev subdomains ensures that search bots never request or index work-in-progress code.

# Pre-production staging environment
# Disallow all bots from all paths
User-agent: *
Disallow: /

Lines 1 and 2 provide descriptive comments for developers inspecting server responses. Line 3 addresses all automated bots, and Line 4 disallows all paths from the root to guarantee zero crawler activity.

The mistakes that cost traffic

Implementing robots.txt incorrectly can inflict catastrophic organic visibility losses across an entire domain. Because search engines evaluate robots.txt rules deterministically, small typographical errors produce sweeping unintended consequences. Understanding the five most dangerous failure modes protects your organic search traffic.

The most widespread historical error is blocking CSS, JavaScript, and font assets. In modern search engine architectures, Googlebot renders web pages using a headless Chromium browser instance to evaluate mobile responsiveness and layout stability. If you disallow /assets/ or /js/, the crawler cannot execute your render tree. Google interprets the broken render as low-quality, mobile-unfriendly content, causing sharp ranking drops.

Accidentally deploying staging robots rules to production represents another frequent catastrophe. When developers push code from staging to production, automated continuous integration pipelines often overwrite production robots.txt with Disallow: /. Within days, search engines halt all crawling across the domain, leading to indexing failures and traffic collapse.

Using robots.txt to hide sensitive user data or secret staging directories introduces severe security risks. Because robots.txt is a publicly accessible file located at /robots.txt, anyone can read it in a web browser. Disallowing paths like /secret-admin/ or /user-tax-records/ essentially publishes a roadmap of high-value targets for malicious actors. Genuine security requires server-level password authentication or IP whitelisting.

Expecting robots.txt to deindex existing URLs represents an operational dead end. If you disallow a page that Google has already indexed, Googlebot cannot fetch the page to discover any on-page noindex directives you might add. The page remains stuck in Google index indefinitely. You must leave the URL crawlable until Googlebot fetches the document and processes the index removal directive.

Finally, file formatting errors like improper character encodings can break rule processing. Saving a robots.txt file with a UTF-8 Byte Order Mark or using non-standard Unicode quotation marks corrupts line parsing. Search engines may fail to read the file, leading to fallback behaviors that disrupt normal crawling schedules.

How to test it

Verifying your robots.txt file before deployment prevents accidental indexing disasters and ensures that search bots interpret your rules as intended. Diagnostic workflows combine first-party search console tools, automated validation scripts, and live command-line HTTP tests.

The most authoritative testing environment is the Robots.txt Tester tool provided inside Google Search Console. This interface allows you to submit specific URL paths and test whether your live robots file blocks or allows Googlebot. It highlights syntax errors, unescaped characters, and logic conflicts in real time.

You can also test your live robots.txt file directly from your local terminal using curl. Inspecting the HTTP headers confirms that your web server returns the correct 200 OK status code and serves the payload with a clean text/plain content type.

curl -I https://example.com/robots.txt

Evaluating the response headers confirms that your server does not serve cached redirect loops or gzip encoding errors. Review our technical guide on HTTP status codes for SEO to understand how response codes impact bot behavior.

What happens when robots.txt is missing or returns an error

Search engines handle missing or failing robots.txt files using conservative fallback policies designed to prevent unintentional data loss. How a search bot responds depends entirely on the HTTP status code returned when requesting /robots.txt. Understanding these edge cases is essential for diagnosing sudden crawling drop-offs.

If your web server returns a 404 Not Found or 410 Gone status code when a crawler requests /robots.txt, the crawler interprets this as full permission. Search engines assume that the website does not maintain an exclusion policy, and all public URLs across the domain are eligible for crawling. A 404 response on robots.txt never halts crawling.

In contrast, if your server returns a 5xx Server Error like 500 Internal Server Error, 502 Bad Gateway, or 503 Service Unavailable, search engines apply a strict fail-safe lockdown. The crawler assumes that you might have intended to block parts of your site, but server failure prevented the file from loading. To prevent crawling restricted areas by mistake, the search engine suspends all crawling across your entire website until the server returns a clean status code.

HTTP 200 OK:        Parse and execute rules normally.
HTTP 404 Not Found: Full permission; all public URLs crawlable.
HTTP 5xx Error:     Complete lockdown; all crawling halted immediately.
Network Timeout:    Treated as 5xx error; crawl requests queued and delayed.

If your server experiences intermittent database connection errors that cause /robots.txt to return 503 status codes, your crawl velocity will collapse overnight. Monitoring server access logs for robots.txt requests is a foundational operational requirement for maintaining search engine visibility.

Frequently asked questions

Is robots.txt required?

A robots.txt file is not strictly mandatory for a website to function or appear in search engine results. If a web server does not contain a robots.txt file, crawlers receive a 404 HTTP status code and assume that all public URLs across the domain are permitted to be crawled without restriction.

Does robots.txt stop a page from appearing in Google?

No, robots.txt does not prevent a URL from appearing in Google search results. It only prevents Googlebot from crawling the page content. If external websites or internal pages link to the disallowed URL using descriptive anchor text, Google can still index the address and display it in search snippets without page text.

What does Disallow: / do?

The directive Disallow: / instructs matching web crawlers not to request any path on that host, effectively blocking the entire website from being crawled. Because the forward slash represents the root path, this prefix match covers every URL beginning with that slash. It is typically used on private staging environments.

Does robots.txt work on subdomains?

A robots.txt file applies strictly to the exact protocol, domain or subdomain, and port where it is hosted. A file placed on example.com has no authority over blog.example.com, shop.example.com, or staging.example.com. Each individual subdomain must host its own separate robots.txt file at its own root path to control crawler access.

Can I use noindex in robots.txt?

No, you cannot use a noindex directive inside robots.txt. Google officially retired support for the experimental noindex directive in robots.txt files in September 2019, and RFC 9309 does not recognize it. To prevent a page from being indexed, serve a meta robots tag or an X-Robots-Tag HTTP response header.

Does Google respect crawl-delay?

Googlebot does not support the crawl-delay directive in robots.txt. If you include crawl-delay in your file, Googlebot parses and ignores that line. To manage Googlebot request frequency, configure crawl rate limits inside Google Search Console or ensure that your server responds with appropriate HTTP status codes when experiencing high load.

Is robots.txt case sensitive?

Directives like User-agent, Allow, and Disallow are case-insensitive, but URL path patterns are strictly case-sensitive. The path /private/ does not match /Private/ or /PRIVATE/. If your server uses uppercase letters in URLs, your robots.txt path definitions must match the exact casing used in links to block crawling properly.

Should I block AI crawlers in robots.txt?

Blocking AI crawlers depends on whether you want your content used in training datasets or retrieval-augmented generative search. Disallowing bots like GPTBot or CCBot prevents automated training scrapers from fetching pages, but it also excludes your content from being cited in emerging AI search engines and answer summaries.