The Complete Guide to AI Crawlers
Every AI web crawler explained: what they do, who operates them, and exactly how to control access via robots.txt. Covers all 17 major AI crawlers active in 2026.
What Are AI Crawlers?
AI crawlers are automated programs that visit websites to collect content for artificial intelligence systems. Every major AI company operates at least one: OpenAI has GPTBot, Anthropic has ClaudeBot, Perplexity has PerplexityBot, Google has Google-Extended, and so on.
These crawlers serve three distinct purposes:
- AI search indexing. OAI-SearchBot, Claude-SearchBot, and PerplexityBot build search indexes for AI-powered search engines. Think of them like Googlebot, but for AI search — these are the visibility gates. Block one and you disappear from that engine's answers.
- Training data collection. GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent and others crawl the web to build training datasets for AI models. Blocking them is a policy choice — it does not affect AI-search visibility.
- User-triggered fetching. ChatGPT-User, Claude-User, and Perplexity-User fetch specific pages when a user asks the AI to look something up mid-conversation.
Search-index and training crawlers respect robots.txt, so you have real control over them. User-fetch agents are the exception: because they act on a direct human request, several (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher) may ignore robots.txt entirely. The challenge is that most website owners don't know these crawlers exist, and many sites accidentally block the search-index bots — the ones that actually matter for visibility — through overly broad robots.txt rules.
17
Major AI crawlers active in 2026
3
Roles: search-index, training, user-fetch
3
Search-index gates that decide AI visibility
Training Crawlers
These crawlers collect content to train AI models. Blocking them prevents the AI from learning from your content in future training runs — a policy choice that does not affect whether AI search engines can show and cite your site.
Training data collection for OpenAI models
Training only. If you block GPTBot, your site won't be included in future model training — but it does NOT remove you from ChatGPT search, which is gated by OAI-SearchBot. Blocking GPTBot while allowing OAI-SearchBot is a common setup: citations without contributing training data.
Training data collection for Claude models
Training only — same idea as GPTBot but for Anthropic. Block it and your content won't appear in future Claude training data, but you stay fully visible in Claude search (that's Claude-SearchBot's job). Note: the old Claude-Web and anthropic-ai tokens are dead — Anthropic removed them from its docs, so rules for them do nothing.
Opt-out token for Gemini training and grounding
Two important distinctions. First, this is NOT Googlebot — blocking Google-Extended has zero effect on Google Search rankings. Second, it does NOT control AI Overviews or AI Mode: those are built from the normal Search index Googlebot crawls. Google-Extended only opts your content out of Gemini app/Vertex AI training and grounding. For AI Overviews control, use nosnippet or data-nosnippet.
Training data collection for Meta AI (Llama)
Meta's main AI training crawler and one of the highest-volume crawlers on the web. It replaced the old FacebookBot token, which Meta has delisted — rules for FacebookBot no longer do anything. Blocking Meta-ExternalAgent keeps your content out of Meta AI training; link previews on Facebook/Instagram are handled separately and are unaffected.
Alexa answers and Amazon AI training
Feeds Alexa's web answers and Amazon's AI models. Blocking it keeps your content out of Alexa responses and Amazon AI training. Amazonbot is also an IndexNow participant, so it picks up page updates quickly on sites that ping IndexNow.
Open web archive used by many AI companies for training
This one flies under the radar but it's arguably the most important training token. Common Crawl is the open web archive that trained the first versions of GPT, Claude, and LLaMA, and it remains the main open training-corpus proxy. Blocking CCBot means you're less likely to appear in any open-source AI model's training data.
Web crawling for ByteDance AI products
ByteDance (TikTok's parent company) runs this crawler for their AI products. Active but undocumented and volatile in behavior, and known for aggressive request volume. Most Western sites block it since ByteDance's AI products primarily serve the Chinese market.
Structured data extraction and knowledge graph building
Diffbot doesn't train a chatbot. Instead, it builds a structured knowledge graph of the web by extracting facts, entities, and relationships from pages. Other AI products then use this data for things like entity recognition.
Opt-out token for Apple AI training
Same logic as Google-Extended: this is a token, not a crawler — the crawling is done by the main Applebot, which has fed Siri and Apple's foundation models since mid-2026. Blocking Applebot-Extended won't affect Safari Suggestions or Siri web results; it only stops Apple from using your content to train Apple Intelligence features.
Browsing Crawlers
These agents fetch pages when a user asks an AI assistant to open a URL. Because they act on a direct human request, several of them ignore robots.txt — treat rules for them as informational, not enforcement.
User-triggered fetch when someone asks ChatGPT to open a URL
When someone pastes your URL into ChatGPT or asks it to read a specific page, this agent fetches it. OpenAI notes robots.txt may not apply to these user-triggered requests, so a Disallow rule here is advisory at best. Treat it as informational, not a visibility control.
User-triggered fetch when someone asks Claude to open a URL
Fires only when a Claude user asks for a specific page — the replacement for the deprecated Claude-Web token. Like other user-fetch agents, it acts on behalf of a human, so treat robots.txt rules for it as informational.
User-triggered fetch for Perplexity conversations
Fetches a page when a Perplexity user explicitly asks for it. Perplexity's own docs say this agent generally ignores robots.txt because it acts on a direct user request — so a Disallow rule here is documentation, not enforcement.
User-triggered fetch for Meta AI assistants
Fetches individual links when Meta AI users ask about them. Because it acts on a direct user request, Meta says it may bypass robots.txt rules. Informational rather than a visibility control.
User-triggered fetch for Le Chat citations
Fetches pages when Le Chat (Mistral's assistant) users ask about them, powering Le Chat's citations. User-triggered rather than bulk crawling.
AI Search Crawlers
These crawlers build search indexes for AI-powered search engines. Blocking them removes your site from AI search results entirely.
Builds the index behind ChatGPT search
This is THE ChatGPT visibility gate. OAI-SearchBot builds the search index ChatGPT consults when it answers with live web results. Block it and your site disappears from ChatGPT search results and citations entirely — no matter what you do with GPTBot. Changes propagate in roughly 24 hours.
Builds the index behind Claude's web search
The Claude visibility gate — Anthropic's equivalent of OAI-SearchBot. It indexes pages for Claude's search answers. Block it and Claude can't surface or cite your site in search results, regardless of your ClaudeBot setting.
Web indexing for Perplexity's AI search engine
Perplexity is an AI search engine that always links to its sources. If you're blocked, you won't show up in Perplexity results at all. Given that Perplexity actually sends traffic back to your site (unlike pure training crawlers), this one's usually worth allowing.
AI Crawler Comparison
| Crawler | Operator | Type | Purpose | Recommendation |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | search | Builds the index behind ChatGPT search | Allow for ChatGPT search visibility |
| GPTBot | OpenAI | training | Training data collection for OpenAI models | Policy choice — doesn't affect search visibility |
| ChatGPT-User | OpenAI | browsing | User-triggered fetch when someone asks ChatGPT to open a URL | Informational — robots.txt may not apply |
| Claude-SearchBot | Anthropic | search | Builds the index behind Claude's web search | Allow for Claude search visibility |
| ClaudeBot | Anthropic | training | Training data collection for Claude models | Policy choice — doesn't affect search visibility |
| Claude-User | Anthropic | browsing | User-triggered fetch when someone asks Claude to open a URL | Informational — user-triggered only |
| PerplexityBot | Perplexity AI | search | Web indexing for Perplexity's AI search engine | Allow for Perplexity citations |
| Perplexity-User | Perplexity AI | browsing | User-triggered fetch for Perplexity conversations | Informational — generally ignores robots.txt |
| Google-Extended | training | Opt-out token for Gemini training and grounding | Policy choice — doesn't affect Search or AI Overviews | |
| Meta-ExternalAgent | Meta | training | Training data collection for Meta AI (Llama) | Policy choice — training only |
| Meta-ExternalFetcher | Meta | browsing | User-triggered fetch for Meta AI assistants | Informational — may ignore robots.txt |
| Amazonbot | Amazon | training | Alexa answers and Amazon AI training | Allow for Alexa answer visibility |
| MistralAI-User | Mistral AI | browsing | User-triggered fetch for Le Chat citations | Allow for Le Chat citations |
| CCBot | Common Crawl Foundation | training | Open web archive used by many AI companies for training | Allow for broad AI training inclusion |
| Bytespider | ByteDance (TikTok) | training | Web crawling for ByteDance AI products | Block if not targeting Chinese AI market |
| Diffbot | Diffbot | training | Structured data extraction and knowledge graph building | Allow for knowledge graph inclusion |
| Applebot-Extended | Apple | training | Opt-out token for Apple AI training | Policy choice — training only |
robots.txt Examples
Allow all AI crawlers (recommended for most sites)
If your robots.txt does not mention a crawler, it is allowed by default. You only need explicit rules if you previously blocked crawlers. This configuration explicitly allows all AI crawlers:
# Allow all AI crawlers # Search-index bots (the visibility gates) User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Training bots User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Amazonbot Allow: / User-agent: CCBot Allow: / User-agent: Applebot-Extended Allow: / # User-fetch agents User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: /
Allow browsing and search, block training
Allow AI assistants to cite your content in real-time, but prevent your content from being used in model training:
# Allow AI search (the visibility gates) User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Allow user-triggered fetching User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / # Block training data collection User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: /
Block all AI crawlers
If you want to prevent all AI access to your content (note: the user-fetch agents at the end may ignore these rules, and legacy tokens like Claude-Web, anthropic-ai, FacebookBot, and cohere-ai are dead — you can delete them from old robots.txt files):
# Block all AI crawlers User-agent: OAI-SearchBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Amazonbot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Diffbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Claude-User Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: MistralAI-User Disallow: /
Check Which AI Crawlers Can Access Your Site
BotView checks your robots.txt against all 17 AI crawlers and shows you exactly what is allowed and blocked. Free scan, 30 seconds.
Frequently Asked Questions
What is an AI crawler?
An AI crawler is a bot that visits websites to collect content for AI systems. Think of it like Googlebot, but instead of indexing pages for search results, AI crawlers are grabbing content to train language models (GPTBot, ClaudeBot), power AI search engines (PerplexityBot), or let AI assistants browse the web in real-time (ChatGPT-User).
How do AI crawlers differ from Googlebot?
Googlebot crawls to index pages for Google Search results. AI crawlers serve different purposes: some collect training data (GPTBot, ClaudeBot, Meta-ExternalAgent), some power AI search engines (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and some fetch pages on a user's request (ChatGPT-User, Claude-User, Perplexity-User). Blocking Googlebot affects your Google rankings. Blocking AI search-index crawlers removes you from that engine's AI answers; blocking training crawlers only affects model training and does not hurt search rankings or AI-search visibility.
Should I block or allow AI crawlers?
It depends on your goals. If you want your content cited by AI assistants, appearing in AI search results, or included in AI training data, allow them. If you are concerned about AI companies using your content without compensation, you can block specific crawlers. Most businesses benefit from AI visibility. You can also selectively allow some crawlers while blocking others.
How do I check which AI crawlers can access my site?
Use BotView to scan your website. We check your robots.txt against all 17 major AI crawlers and show you exactly which ones are allowed and which are blocked. The scan is free and takes about 30 seconds.
Does blocking AI crawlers affect my Google ranking?
No. AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) are completely separate from Googlebot. Blocking AI crawlers has zero effect on your Google Search rankings. The only exception is Google-Extended, which controls AI training usage but not search indexing — blocking it does not affect your Google rankings either.
What is the difference between training crawlers and browsing crawlers?
Training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent) collect content in bulk to train AI models — blocking them is a policy choice that doesn't affect AI-search visibility. Search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the live indexes behind AI search answers — blocking these DOES remove you from those engines. User-fetch agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a specific page when a user asks; several of them ignore robots.txt because they act on a direct human request.
How do I allow all AI crawlers at once?
If your robots.txt doesn't mention a crawler at all, it's allowed by default. So the simplest approach is to just not block them. The problem comes if you have a broad rule like 'User-agent: * / Disallow: /' that blocks everything. In that case, you need to add specific Allow rules for each AI crawler you want to let through.
Can AI crawlers see JavaScript-rendered content?
Most can't. They see the raw HTML your server sends back, not the final page after JavaScript runs. If your site is built with React, Vue, or Angular and renders content client-side, AI crawlers might see a blank page. This catches a lot of people off guard. BotView shows you a side-by-side comparison of what humans see vs what crawlers see, so you can spot this instantly.
Related Guides
- Check ChatGPT visibility — Is GPTBot blocked?
- Check Claude visibility — Is ClaudeBot blocked?
- Check Perplexity visibility — Is PerplexityBot blocked?
- Check Copilot visibility — Is your site cited by Microsoft Copilot?
- GPTBot robots.txt guide — Detailed GPTBot configuration
- How to appear in AI search — Step-by-step guide
- What is GEO? — Generative Engine Optimization explained