AI Crawlers List: How GPTBot, ClaudeBot, and CCBot Work

AI crawlers are automated software programs operated by artificial intelligence laboratories to extract web content for model training, synthetic dataset generation, and real-time retrieval-augmented generation. Unlike conventional search engine spiders that deliver referral clicks, AI scrapers ingest raw text to compile generative tokens. Website operators can inspect, verify, and restrict these autonomous agents using user-agent directives declared in their robots.txt configuration.

What are AI web crawlers and how do they differ from search engines?

AI web crawlers are autonomous HTTP clients deployed by machine learning organizations to harvest textual and multimodal assets from public web servers. While traditional search spiders index content to display ranked blue links in response to keyword queries, artificial intelligence bots extract content to train deep neural networks or ground conversational answers. This divergence in operational purpose fundamentally alters the economic exchange between digital publishers and crawling software.

Traditional search engines operate on a reciprocal value exchange. When Googlebot or Bingbot crawls your web pages, your server expends bandwidth and compute resources to process the incoming HTTP requests. In exchange, the search engine indexes your content and directs targeted organic visitors back to your domain. This dynamic provides webmasters with a direct incentive to welcome search spiders and maintain open access across their site architecture.

AI training crawlers break this historical exchange by consuming server capacity without providing referral traffic in return. When an AI bot ingests your documentation, blog posts, or academic research, the material is tokenized, vectorized, and compressed into neural network weights. When end users later ask questions related to your content, the generative language model synthesizes an answer directly inside the chat interface. Because users receive their complete answer immediately, they rarely click through to visit the primary source website.

Traditional Search Engine Pipeline:
Website Content ──► Googlebot Fetch ──► Search Index ──► SERP Result ──► Referral Click to Website

AI Training and Synthesis Pipeline:
Website Content ──► AI Bot Scrape ──► Tokenized Corpus ──► Model Weights ──► Direct Chat Answer (No Click)

The underlying technical mechanics of an AI crawler mirror those of traditional automated web crawlers. Both systems resolve DNS records, negotiate TLS handshakes, parse robots.txt policies, issue HTTP GET requests, and extract text and outbound links. However, AI crawlers often operate with far higher concurrency and less sensitivity to host response times. While commercial search engines deliberately tune their fetch frequency to prevent server degradation, uncalibrated AI scrapers can overwhelm origin servers with hundreds of simultaneous connections.

Furthermore, AI crawlers exhibit varying architectural scopes. Some bots act as broad web harvesters gathering trillions of tokens across millions of domains for initial model training runs. Other bots act as precision retrieval agents dispatched on demand when an individual user submits a prompt requiring current factual information. Understanding which category a specific crawler belongs to is essential for crafting an effective infrastructure policy.

Comprehensive list of AI crawler user agents and operators

Managing automated traffic requires identifying the exact user-agent string that each artificial intelligence organization uses to identify its crawling software. The following reference table lists the most prominent AI crawlers active across the web, their operating entities, their primary functional purposes, and their compliance status with robots.txt standards.

Table last verified: September 2026.

OpenAI maintains three separate bot identities to give publishers granular control over how their content is accessed. GPTBot crawls autonomously across the web to gather training data for upcoming models like GPT-5. ChatGPT-User operates strictly in response to an active ChatGPT user asking the assistant to view a specific link or perform a live web search. OAI-SearchBot builds the search index that powers OpenAI Search, linking directly to source citations without using the scraped text for model training.

Anthropic operates a similar dual-agent architecture. ClaudeBot gathers general web documents to expand the broad capabilities of the Claude model family. When an end user prompts Claude to inspect an explicit URL, the system dispatches Claude-Web to fetch only that targeted resource. Allowing Claude-Web while disallowing ClaudeBot permits users to analyze your pages inside the assistant while protecting your proprietary text from pre-training ingestion.

ByteDance operates Bytespider to support Douyin, TikTok algorithms, and proprietary machine learning models. Webmasters frequently report Bytespider as one of the most aggressive crawlers on the internet. In many instances, network logs indicate that Bytespider nodes ignore robots.txt crawl-delay settings and continue hammering origin servers even after receiving HTTP 429 status codes.

Google and Apple have introduced standalone tokens that operate alongside their commercial search bots. Google uses Google-Extended as a dedicated switch. Disallowing Google-Extended prevents Google from using your site content to train Gemini and Vertex AI models, while leaving your visibility in traditional search results and standard Googlebot crawling completely unaffected. Similarly, Applebot-Extended controls training rights for Apple Intelligence without compromising presence in Spotlight search.

Pre-training scrapers versus real-time RAG grounding bots

The critical operational distinction in modern AI crawling lies between bulk pre-training scrapers and real-time retrieval-augmented generation (RAG) grounding bots. These two bot categories possess completely different request profiles, bandwidth consumption patterns, and commercial outcomes for website operators.

Pre-training crawlers operate asynchronously during massive, multi-month computational cycles. When an artificial intelligence research firm begins developing a new foundation model, it requires trillions of unique text tokens spanning billions of web pages. Crawlers like GPTBot, ClaudeBot, and CCBot scour the web systematically, traversing link graphs and consuming immense volumes of data over short periods. These crawlers focus on volume, downloading every accessible page, stylesheet, and article across your domain.

Pre-Training Scraper Architecture:
Scheduler ──► Multi-threaded Worker Pool ──► Fetch Trillions of Pages ──► Raw Text Tokenization ──► Training Dataset

Real-Time RAG Grounding Architecture:
User Prompt: "Compare laptop warranties" ──► Query Rewriter ──► Live Bot Fetch (1-5 URLs) ──► In-Context Synthesis ──► User Response

Because pre-training crawls occur offline, the pages downloaded today may not appear in an active AI model for six to twelve months. Furthermore, once pre-training completes, the downloaded text is absorbed into billions of numerical parameters. There is no ongoing network connection between the deployed model and your web server, and the resulting AI responses rarely cite your URL or pass clickable attribution to your brand.

Real-time RAG bots operate synchronously under strict latency constraints. When an end user queries an AI search engine like Perplexity or ChatGPT Search, the model recognizes that its internal training data is outdated or insufficient to answer the query accurately. The system immediately executes live search queries, identifies three to ten authoritative URLs, and dispatches a lightweight fetcher like PerplexityBot or OAI-SearchBot to download those specific documents.

RAG fetchers do not scrape your entire directory tree. Instead, they retrieve only the targeted HTML document, extract the most relevant passages, inject those excerpts into the model prompt context window, and synthesize a referenced answer with clickable footnote links. For publishers seeking digital visibility, RAG crawlers represent a valuable channel. Blocking pre-training bots while welcoming RAG agents allows you to preserve your data assets while earning prominent visibility in conversational search engines and modern AI overview answers.

How to verify genuine AI crawler IP addresses and reverse DNS

Because AI bot user-agent strings are widely known, malicious scrapers, commercial copycats, and vulnerability scanners frequently forge headers like GPTBot or ClaudeBot to bypass security barriers. Allowing traffic based solely on the incoming User-Agent HTTP header creates severe security vulnerabilities. Engineering teams must verify crawler authenticity using automated reverse DNS lookups and published IP range tables.

The standard industry procedure for verifying legitimate automated clients is reverse DNS lookup followed by forward DNS validation. This two-step handshake prevents IP spoofing because an attacker cannot forge hostname records within a provider’s authoritative domain.

Two-Step DNS Verification Process:
Step 1: Reverse Lookup (PTR)
Client IP: 20.171.207.200 ──► Reverse DNS (PTR) ──► crawler.openai.com

Step 2: Forward Lookup (A/AAAA)
Host: crawler.openai.com ──► Forward DNS (A) ──► 20.171.207.200 (Matches Client IP)

To execute this verification manually in a Linux terminal, use the host utility to inspect the incoming IP address:

# Perform reverse DNS lookup on suspicious crawler IP
host 20.171.207.200

# Expected output confirms OpenAI hostname
200.207.171.20.in-addr.arpa domain name pointer gptbot-20-171-207-200.openai.com.

# Verify forward resolution matches original IP
host gptbot-20-171-207-200.openai.com

# Expected output confirms valid address
gptbot-20-171-207-200.openai.com has address 20.171.207.200

If the domain name pointer does not end in .openai.com, or if the forward resolution yields a different IP address, the request is an imposter and should be rejected immediately with an HTTP 403 Forbidden status code.

Anthropic does not currently publish individual reverse DNS PTR records for all crawler nodes. Instead, Anthropic publishes a structured JSON file containing all official IP prefixes allocated to ClaudeBot. System administrators can automate firewall rules by syncing their edge access policies with Anthropic’s CIDR ranges:

{
  "creationTime": "2026-09-01T12:00:00.000Z",
  "prefixes": [
    { "ipv4Prefix": "160.79.104.0/23" },
    { "ipv6Prefix": "2607:6bc0::/32" }
  ]
}

Cloudflare and other enterprise web application firewalls (WAF) provide built-in managed challenge rules for verified bots. Enabling “Verified Bots” allows legitimate search engines and verified AI crawlers through while automatically challenging or dropping forged user-agent traffic.

Controlling AI bots using robots.txt directives and HTTP headers

The primary mechanism for governing artificial intelligence scrapers is the robots.txt protocol, formally standardized as RFC 9309. Compliant AI laboratories program their crawlers to fetch and evaluate /robots.txt before requesting any document from a website. By crafting targeted directive blocks, webmasters can allow search engine spiders while selectively restricting generative training bots.

To block pre-training crawlers while maintaining normal search engine visibility and allowing real-time AI citations, implement the following configuration:

# Allow conventional search crawlers full access
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Prevent foundation model pre-training ingestion
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: CCBot
Disallow: /

# Allow real-time search and conversational citation fetchers
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

In addition to user-agent tokens, web publishers can signal machine-readable usage permissions using HTTP response headers. The X-Robots-Tag header provides page-level and asset-level control for documents such as PDFs, images, and data feeds:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noai, noimageai

The emerging noai and noimageai directives instruct compliant models not to incorporate the delivered binary or textual content into artificial intelligence datasets. While not every open-source scraper honors custom headers, major commercial platforms increasingly recognize these explicit declarations.

Publishers can also supply explicit guidance for automated language models by serving an llms.txt markdown file in their site root directory. This structured plain-text file lists clean, documentation-focused markdown paths designed for direct machine ingestion, steering RAG systems toward authoritative reference endpoints.

Managing bandwidth costs and server load from aggressive scrapers

Aggressive web scrapers pose a significant financial and operational risk to web infrastructure. Because many AI scraping scripts run multi-threaded workers across cloud providers like Amazon Web Services, Microsoft Azure, and DigitalOcean, a single unthrottled crawl can initiate tens of thousands of requests per minute. This activity inflates bandwidth egress bills, consumes database connection pools, and degrades page responsiveness for real human users.

When unmanaged bots flood your infrastructure, they deplete server capacity, causing the same host degradation problems documented in our analysis of crawl budget limits. If your web server is busy responding to scrapers, genuine search engine crawlers may receive timeouts and back off, damaging your organic search performance.

Server Load Impact of Unmanaged Scraping:
Aggressive Scrapers (500 req/sec) ──┐
                                     ├──► Origin Server CPU: 98% ──► 503 Overload ──► Googlebot Drops Crawl Rate
Genuine Search Spiders (20 req/sec) ─┤
Human Visitors (50 req/sec) ─────────┘

To protect origin infrastructure, engineering teams implement rate-limiting policies at the edge server or load balancer level. For instance, an Nginx reverse proxy configuration can restrict request velocity for designated AI user-agents:

# Define rate limiting zone based on client IP
limit_req_zone $binary_remote_addr zone=ai_bot_limit:10m rate=5r/s;

# Apply rate limiting to aggressive AI crawler tokens
map $http_user_agent $is_ai_bot {
    default 0;
    "~*Bytespider" 1;
    "~*CCBot" 1;
    "~*Diffbot" 1;
}

server {
    listen 443 ssl http2;
    server_name example.com;

    location / {
        if ($is_ai_bot) {
            limit_req zone=ai_bot_limit burst=10 nodelay;
        }
        try_files $uri $uri/ /index.php?$query_string;
    }
}

If an aggressive bot like Bytespider systematically ignores robots.txt directives and bypasses rate limits, edge rules should return an HTTP 403 Forbidden or HTTP 429 Too Many Requests response code. Cloudflare edge workers can also be configured to drop connections from automated clients that fail managed challenges, preserving origin bandwidth entirely.

The role of Common Crawl and CCBot in foundation model training

Common Crawl is a 501(c)(3) non-profit organization that maintains an open repository of web crawl data accessible to anyone for research and commercial development. Founded in 2007, Common Crawl periodically releases massive snapshots of the public web containing billions of web pages and petabytes of raw textual data. The crawler responsible for collecting this corpus is identified by the user-agent token CCBot.

Common Crawl serves as the foundational data backbone for the entire generative artificial intelligence ecosystem. Building a custom web crawler to index billions of international domains requires millions of dollars in bandwidth, proxy infrastructure, storage clusters, and engineering overhead. Rather than building proprietary crawlers from scratch, most artificial intelligence companies, academic institutions, and open-source projects rely on Common Crawl archives.

The Generative AI Supply Chain:
Public Internet ──► CCBot Scraping ──► Common Crawl Archives (WARC Files)
                                                    │
             ┌──────────────────────────────────────┴──────────────────────────────────────┐
             ▼                                      ▼                                      ▼
     Llama (Meta)                           GPT-3/4 (OpenAI)                       Mistral / Falcon

Published research papers reveal the immense influence of Common Crawl. Meta’s open-weights model family Llama, OpenAI’s GPT-3 and GPT-4, and models from Mistral AI all acknowledge Common Crawl as a core component of their pre-training corpora. Even when an AI organization does not send its proprietary bot to your server, your content has likely entered model training datasets through Common Crawl snapshots.

CCBot strictly adheres to the robots.txt standard. If you declare a disallow directive for CCBot in your robots.txt file, the crawler will not index your domain during future crawl cycles:

User-agent: CCBot
Disallow: /

It is crucial to understand the temporal reality of Common Crawl. Blocking CCBot prevents your website from appearing in future monthly crawl releases. However, historical Common Crawl archives are immutable and stored permanently on Amazon S3 public datasets. Text scraped from your website in previous years remains accessible to AI researchers and cannot be revoked retroactively from existing datasets.

Legal, ethical, and architectural considerations for AI data access

The proliferation of automated AI scraping has ignited profound legal and commercial debates regarding copyright, fair use, intellectual property, and fair internet architecture. At the center of this conflict is whether scraping copyrighted web material to train generative models constitutes legal fair use under United States copyright law.

Artificial intelligence companies argue that training neural networks is transformative. In copyright jurisprudence, transformative use occurs when an original work is converted into something with a distinctly different purpose or character. AI laboratories contend that their systems do not copy or redistribute original text verbatim; instead, they analyze statistical relationships between words to learn language concepts, syntax, and world knowledge.

Publishers and content creators argue that generative models compete directly with the very source material used to train them. When an AI chatbot generates exhaustive summaries, code snippets, or analytical reports based on copyrighted works without compensation or traffic delivery, it directly harms the market value of the original publications. Numerous high-profile copyright infringement lawsuits filed by news organizations, authors, and visual artists are currently making their way through federal courts to resolve this dispute.

From a systems architecture perspective, webmasters must establish a clear, intentional data governance policy. Allowing unregulated scraping devalues proprietary databases, exposes internal site structures, and drains computing budgets. Conversely, blocking every automated agent cuts your brand off from emerging generative search engines, conversational answer engines, and assistant ecosystems.

A forward-thinking architecture relies on balanced, tiered access policies:

1. Permit Traditional Search: Maintain unrestricted access for search engines like Google and Bing to protect organic discoverability and revenue.
2. Permit Real-Time RAG Bots: Allow verified citation agents like OAI-SearchBot and PerplexityBot to secure referral links in generative search results.
3. Restrict Mass Pre-Training Harvesters: Disallow high-volume bulk scrapers like GPTBot, ClaudeBot, and Bytespider to prevent uncompensated data extraction.
4. Block Malicious and Non-Compliant Bots: Enforce WAF rate limits and managed challenges against crawlers that ignore robots.txt standards or spoof user-agent headers.

Establishing these technical boundaries ensures that your content supports your organization’s business objectives while maintaining robust security and performance across the modern search and AI ecosystem explained throughout Search Engine Basics.

Frequently asked questions

What is the difference between GPTBot and ChatGPT-User?

GPTBot is an automated crawler that harvests web pages to build pre-training datasets for OpenAI foundation models. In contrast, ChatGPT-User is an on-demand fetcher that retrieves specific web pages in real time only when an active ChatGPT subscriber prompts the system to view a link.

How do I block all AI crawlers in robots.txt?

You can block major AI crawlers by adding explicit disallow blocks for known user agents such as GPTBot, ClaudeBot, CCBot, and Bytespider in your robots.txt file. Because AI bots do not share a single universal token, each crawler user agent must be specified separately.

Does blocking ClaudeBot hurt my Google search rankings?

Blocking ClaudeBot has zero effect on your Google search rankings. ClaudeBot is operated by Anthropic solely to collect machine learning training data. Google uses its own separate Googlebot crawler to index content, calculate search rankings, and generate organic search engine result pages.

Why does CCBot crawl my website if I don’t use AI?

CCBot crawls public websites on behalf of Common Crawl, a non-profit foundation that builds an open digital archive of the internet. AI labs and academic researchers download these public snapshots to train models without needing to build and run their own crawling infrastructure.

Can AI crawlers bypass robots.txt directives?

Compliant AI crawlers from established organizations like OpenAI, Anthropic, and Common Crawl strictly respect robots.txt instructions. However, unregulated scrapers and rogue scraping scripts can ignore robots.txt completely. Enforcing edge firewall rules and IP rate limits is required to stop non-compliant bots.

What is Google-Extended and what does it control?

Google-Extended is a specific user-agent token that allows webmasters to manage whether their content is used to train Google generative AI systems, including Gemini and Vertex AI APIs. Blocking Google-Extended prevents AI model training while preserving normal indexing and visibility in Google search.

How can I verify that a GPTBot request is really from OpenAI?

You can verify authentic GPTBot requests by performing a reverse DNS lookup on the incoming IP address to confirm it resolves to an openai.com domain. You then run a forward DNS lookup on that hostname to ensure it matches the original client IP address.

Will blocking AI crawlers prevent my site from appearing in AI summaries?

Blocking pre-training bots like GPTBot will not prevent your site from appearing in real-time AI summaries if you permit RAG fetchers like OAI-SearchBot and PerplexityBot. However, blocking all automated retrieval bots will prevent conversational search engines from displaying current citations to your content.

Sources

• OpenAI Documentation: Overview of OpenAI Crawlers
• Anthropic: What is ClaudeBot and how do I block it?
• Common Crawl: Frequently Asked Questions and CCBot
• IETF RFC 9309: Web Robots Exclusion Protocol