AI Visibility

AI Crawler Access: Controlling GPTBot, ClaudeBot & the Rest

A wave of AI crawlers now visits your site for different reasons, and deciding which to allow or block is a real business trade-off between protecting your content and staying visible in AI answers.

By Vega, NYFTY Labs AI Content Engine · AI-drafted, human-reviewed

The web used to have one crawler that really mattered: Googlebot. Now a growing set of AI crawlers visits your site for different reasons, and deciding who gets access is a deliberate business choice, not an accident. The essential move is to separate purpose from owner: some bots collect data to train models, some index your pages for AI search answers, and others fetch pages live because a user asked. Blocking one is not blocking the others. This is who the real 2026 crawlers are, how robots.txt lets you state preferences (and why it is only a request), what Cloudflare changed, and the honest trade-off between protecting content and staying visible in AI answers.

AI crawlers do three different jobs, training, search-indexing, and live user-initiated fetch, and meet your site at a gate: robots.txt states a preference (voluntary), while Cloudflare Pay Per Crawl can enforce access with an HTTP 402 toll.

Three jobs: training, search-index, and live fetchers

The most important distinction among AI crawlers is not the company but the purpose. Training or data-collection crawlers gather content to help build or improve AI models. Search-index crawlers catalog your pages so you can be surfaced and cited in an AI product's search-style answers. Live, on-demand fetchers retrieve a page in real time because a user asked a question or clicked a link inside an AI product. These serve different ends, and blocking one is often not the same as blocking the others. Many vendors expose separate tokens per purpose, but this is not universal: Meta-ExternalAgent is explicitly multipurpose, and Google provides no robots.txt token that blocks its Search AI features while preserving classic Search indexing - so the three purposes cannot always be controlled independently. Where they are separated, you can refuse training use while still allowing the search indexing and the live fetch that can put you inside an AI answer.

The key distinction is purpose: training, search-indexing, and live on-demand fetch are different jobs.

The real 2026 user-agents to know

Vague advice to "block AI bots" is useless without names. OpenAI: GPTBot (training), OAI-SearchBot (search-style indexing and surfacing), ChatGPT-User (live fetch) - each a separate robots.txt token. Anthropic's documented crawlers: ClaudeBot (training), Claude-SearchBot (navigates and analyzes the web to improve Claude's search-result quality), Claude-User (user-initiated live fetch); the older anthropic-ai token is deprecated, and Anthropic does not publish a separate robots.txt token for Claude Code (its URL fetching is not independently controllable via a dedicated user agent). Others: PerplexityBot (search indexing) and Perplexity-User (live fetch); CCBot (Common Crawl, a large open dataset many models have drawn on); Google, where Googlebot still handles both Search and Google's AI features, Google-Extended is an AI-training opt-out token (not a distinct crawler), and Google-CloudVertexBot is a separate crawler that fetches sites at the site owner's request when a customer is building Vertex AI agents or grounding; Applebot with the Applebot-Extended opt-out token; Amazonbot, Bytespider (ByteDance), Meta-ExternalAgent (Meta's multi-purpose crawler, including AI training as well as indexing and product improvement) and facebookexternalhit (Meta link previews). These are among the major agents and the names you reference in your rules, but the roster is not exhaustive and keeps changing, so check each provider's current documentation.

Each vendor exposes separate robots.txt tokens per purpose; the older anthropic-ai token is deprecated.

robots.txt states preferences, but it is voluntary

robots.txt, a plain text file at your domain root, has governed crawler behavior for decades under the Robots Exclusion Protocol, formalized as RFC 9309 in September 2022. You name a user-agent and allow or disallow paths, so you can disallow GPTBot while leaving Googlebot untouched. The essential caveat: robots.txt is a request, not enforcement. Well-behaved crawlers generally honor it and major providers publish support for their tokens, but nothing in the protocol physically stops a bot from ignoring it. It is not a security control. A further limit: robots.txt reliably governs autonomous crawlers (GPTBot, ClaudeBot, CCBot), but user-triggered live fetchers vary. Anthropic documents that Claude-User respects it; ChatGPT-User is a published, identifiable user-agent token, but how user-initiated fetches interact with robots.txt has been an evolving area - consult OpenAI's current documentation for the authoritative rule, and do not assume a user-triggered fetch is reliably blocked by robots.txt alone. By contrast, Cloudflare reported in August 2025 that Perplexity kept reaching blocked content via undeclared user-agents and rotating IPs and de-listed it as a verified bot (Perplexity disputes this). So do not count on robots.txt to block user-triggered fetches everywhere.

robots.txt (RFC 9309) states preferences per token, but honoring it is voluntary, not enforced.

Cloudflare added real enforcement beyond a polite request

Because robots.txt is only voluntary, infrastructure-level controls matter. Cloudflare, which fronts a large share of the web, has moved to give site owners stronger controls over AI crawlers, including blocking many AI bots by default for new sites. On July 1, 2025 it launched, in private beta, a pay-per-crawl model that uses the HTTP 402 (Payment Required) response so operators can charge AI companies for access rather than serving them free, and its broader AI Crawl Control dashboard later reached general availability. At the Pay Per Crawl launch, Cloudflare described a move toward usage-based ('pay per inference') compensation as a future direction; no date has been announced - check Cloudflare's current documentation for status. Specifics here move fast, but the durable point is that access shifts from a request the site makes to an enforced gate the network operator controls, a real change from the robots.txt-only era.

Cloudflare Pay Per Crawl uses HTTP 402 so operators can charge AI companies rather than serve them free.

The trade-off, and where llms.txt fits

The decision is a genuine business trade-off. Blocking an identified training crawler limits future collection by that bot (it does not retract copies already collected, licensed, obtained via third-party datasets, or gathered by disguised agents), which matters most for publishers and original-content businesses, but blocking live and search-index fetchers can directly cut your chances of being retrieved and cited in AI answers. There is no universally correct setting; decide per purpose based on whether visibility or protection is more valuable to you. And do not overstate llms.txt: it is a proposed, unofficial convention no major provider must honor, an optional signal, not a control mechanism or a substitute for robots.txt.

A genuine trade-off decided per purpose: protection versus the chance to be retrieved and cited.

Key takeaways

Separate purpose from owner: training/data-collection crawlers, search-index crawlers, and live user-triggered fetchers are three different jobs, and blocking one does not block the others.
Use the real, current user-agent names rather than vague "AI bot" advice, e.g. GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot, Claude-User (Anthropic; the older anthropic-ai token is deprecated, and Anthropic publishes no separate robots.txt token for Claude Code); PerplexityBot, Perplexity-User; CCBot; Googlebot, Google-Extended and Google-CloudVertexBot; Applebot and Applebot-Extended; Amazonbot, Bytespider, Meta-ExternalAgent, facebookexternalhit; this list is not exhaustive and evolves, so check each provider's docs.
robots.txt (Robots Exclusion Protocol, RFC 9309, September 2022) states per-user-agent preferences but is voluntary: cooperative bots generally honor it, yet it is not enforcement or security, and it reliably governs autonomous crawlers more than user-triggered fetchers; Anthropic documents that Claude-User honors it, while ChatGPT-User is a published, identifiable token whose robots.txt interaction is an evolving area (check OpenAI's current docs; do not assume a user-triggered fetch is reliably blocked by robots.txt alone), and Cloudflare reported Perplexity's agents evading blocks in August 2025.
Cloudflare's default AI-bot blocking and its pay-per-crawl model (private beta from July 1, 2025, using HTTP 402, with usage-based 'pay per inference' compensation described at launch as a future direction and no date announced) are real infrastructure-level enforcement beyond voluntary robots.txt, worth considering if content protection is serious.
The block-versus-allow decision is a real trade-off between protecting content and staying citable in AI answers; decide per purpose, and treat llms.txt as an optional, unofficial signal only.

← More guides

FAQ

Questions, answered.

A training or data-collection crawler (such as GPTBot or ClaudeBot) gathers content used to help build or improve AI models. A live fetcher (such as ChatGPT-User or Claude-User) retrieves a page in real time because a user asked a question or clicked a link inside an AI product. A third category, search-quality crawlers (such as OAI-SearchBot, Claude-SearchBot, or PerplexityBot), navigates and analyzes the web so you can be surfaced and cited in AI search-style answers. They serve different purposes, and blocking one does not block the others, so you can, for example, restrict training use while still allowing the search indexing and live fetches that can get you cited. One caveat: robots.txt is a dependable control for well-behaved autonomous crawlers, but some user-triggered fetchers do not reliably honor it. Anthropic documents that Claude-User respects robots.txt; ChatGPT-User is a published, identifiable user-agent token, but how user-initiated fetches interact with robots.txt has been an evolving area - consult OpenAI's current documentation for the authoritative rule, and do not assume a user-triggered fetch is reliably blocked by robots.txt alone. Cloudflare reported in August 2025 that Perplexity's agents evaded blocks, so blocking user-triggered fetchers is less dependable.

It expresses a request that well-behaved crawlers generally honor, and OpenAI publishes support for its user-agent tokens and separate GPTBot, OAI-SearchBot and ChatGPT-User controls. But robots.txt is voluntary by design; it is part of the Robots Exclusion Protocol (formalized as RFC 9309 in September 2022), which documents the convention without enforcing it. Nothing in the protocol physically prevents a non-compliant bot from ignoring the file. Blocking GPTBot governs OpenAI's training crawler specifically; it does not, by itself, affect OAI-SearchBot or ChatGPT-User, which you control with their own tokens. So robots.txt is the right first tool for stating preferences, but it is not a security control or a guarantee.

They are control tokens rather than separate crawlers. Google-Extended, per Google, lets you opt out of having your content used to train its generative AI models (such as Gemini and its Vertex AI generative features); Google states that using it does not affect how Googlebot indexes or ranks your pages in Search. Note an important limit: because Googlebot is a single crawler serving both Search and Google's AI features, Google-Extended controls generative-model training but does not remove your pages from AI Overviews inside Google Search. Applebot-Extended plays a similar opt-out role for Apple's AI-model training, layered on top of the normal Applebot search crawler so blocking it does not remove you from Apple's search features. Both let you separate "index me for search" from "use me for AI training," which is exactly the kind of purpose-based control this topic is about. (Google also runs a separate Google-CloudVertexBot that fetches sites at the site owner's request when a customer is building Vertex AI agents or grounding, which is distinct from the Google-Extended token.)

It is a genuine business trade-off with no universal answer. Blocking an identified training crawler limits future collection by that bot, which matters most for publishers and original-content businesses, though it does not retract copies already collected, licensed, obtained via third-party datasets, or gathered by disguised agents. But blocking live fetchers and search-index crawlers in particular can reduce your chances of being retrieved and cited when users ask relevant questions in AI tools. The practical approach is to decide per purpose: many businesses allow live and search-style fetchers for visibility while restricting or negotiating around bulk training crawlers.

It is optional and experimental, not required. The llms.txt file is a proposed, community-originated convention, a Markdown file pointing AI systems to your key content, similar in spirit to a sitemap. It is low-cost to adopt, but it is not an official standard and no major AI provider is obligated to honor it. Treat it as a bonus signal if you want, not as a control mechanism and not as a replacement for robots.txt or for deliberate crawler-access decisions.

Want this working for your brand?

Start your project Try the free tools