Introduction
You ask ChatGPT about your own product and it recommends a competitor. Perplexity cites three other sites for a question your page answers best. Or you paste your link into Claude and it says it cannot open the page. Advice for this usually starts with content, AI SEO and generative engine optimization, but there is a plainer question first: can the machine get the page at all?
What are AI crawlers? AI crawlers are the programs AI companies use to fetch web pages. Some copy pages to train models. Some build the search index behind ChatGPT search, Claude, Perplexity and Google's AI Overviews. Others open one page at the moment a user asks about it. Each has its own name in the User-Agent header and its own entry in robots.txt, and a site can let one in while shutting out another, often without meaning to.
Below: where each AI assistant gets its pages, including the common question of whether ChatGPT uses Bing, how the three kinds of AI bots work, why AI cannot see a website, how to block training without leaving AI search, and how to check what AI crawlers receive.
Where do ChatGPT, Claude, Gemini and Perplexity get web pages?
An AI assistant has two ways of knowing about a page. The first is training: pages copied months ago became part of the model, without links and without your latest changes. The second is live search: during the conversation the assistant looks pages up in a search index, reads them and cites them with a link. When people talk about AI search visibility or how to show up in ChatGPT, they mean the second path, an answer that names and links your page. Each operator documents the crawlers behind that path, and those pages are the best guide to what each bot does.
Does ChatGPT use Bing or Google?
OpenAI names its own crawler for this job. In its words, OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features", and sites that block it "will not be shown in ChatGPT search answers, though can still appear as navigational links". Whether a site appears in ChatGPT search starts with whether OAI-SearchBot can read it.
How does ChatGPT search the web? According to OpenAI, ChatGPT can turn a request "into one or more search queries, retrieve relevant results, and use those results to generate an answer with links to sources". When it launched ChatGPT search in October 2024, OpenAI said the feature uses third-party search providers as well as content supplied by its partners.
Its help pages name one of those providers. For business and education workspaces, ChatGPT "may share disassociated search queries with the Bing search engine to return web results". OpenAI also keeps a search index of its own: its HIPAA-eligible workspaces for healthcare use only that index, and OpenAI does not send their queries to "third-party search providers (e.g., Bing)". Microsoft had announced in May 2023 that it was "bringing Bing to ChatGPT as the default search experience". Google is not named on any of these pages.
So does ChatGPT use Bing or Google, and what search engine does ChatGPT use? Bing, together with OpenAI's own index, which OAI-SearchBot fills. A site that wants to appear in ChatGPT search needs both: OAI-SearchBot allowed in robots.txt and the site indexed by Bing.
Google AI Overviews, AI Mode and Gemini
Google does not run a separate AI crawler for search. AI Overviews and AI Mode are built on the normal Google Search index, and Google says there are "no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary". A page shows up as a supporting link only if it "must be indexed and eligible to be shown in Google Search with a snippet". The control is Googlebot: blocking Googlebot in robots.txt takes a site out of Google Search, AI features included.
Google-Extended is a robots.txt token with no requests of its own. It decides whether pages Google crawls may be used to train future Gemini models and to ground answers in the Gemini apps. Google states that "Google-Extended does not impact a site's inclusion in Google Search", so disallowing it does not remove a site from AI Overviews.
Microsoft Copilot and Bing
Bing has no separate AI crawler either. Bingbot collects pages for Bing, and Microsoft describes the same crawled content as the source of "web-grounded responses in Bing, Copilot, and Microsoft 365". A site that blocks Bingbot drops out of Bing, and its pages are no longer collected for those answers.
On October 30, 2026 Microsoft announced separate AI preferences: ai-grounding:disallow in a robots meta tag or HTTP header keeps a page out of grounded answers, and a shared robots.txt group described below opts a site out of training. They apply from February 1, 2027, and "these settings govern use of content, not crawl access".
Claude
Anthropic runs three bots and describes each one. ClaudeBot collects content that may be used for training. Claude-SearchBot "navigates the web to improve search result quality for users". Claude-User fetches a page when a person asks Claude about it. Anthropic says its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt", and unlike most operators it makes no exception for the user-triggered one.
Perplexity
PerplexityBot is "designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models", so Perplexity advises allowing it. Perplexity-User handles user actions, and "since a user requested the fetch, this fetcher generally ignores robots.txt rules".
Apple, Meta, Amazon, DuckDuckGo and Mistral
The other operators split the work the same way. Applebot powers search in Spotlight, Siri and Safari, while Applebot-Extended is only a token for opting out of training. Meta, Amazon and Mistral each run one bot for search and another for training, and DuckDuckGo says the pages DuckAssistBot reads for its AI-assisted answers are not used to train AI models.
How do AI crawlers work?
All of these AI bots fall into three kinds, and the kind decides what blocking them costs.
- Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot) copy pages that may end up in a model's training data. Blocking them keeps future content out of training and does not remove the site from any search.
- Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot and Bingbot for the AI features of Google and Microsoft) build the index that answers draw on. Blocking them is what takes a site out of AI answers.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User) open one page because a person asked. Several operators say robots.txt may not apply to them.
That settles the common question of OAI-SearchBot vs GPTBot. In OpenAI's words, "Each setting is independent of the others", and a site can allow OAI-SearchBot to appear in search while disallowing GPTBot so its content is not used for training.
Is ChatGPT a web crawler? ChatGPT itself is the app. OpenAI runs crawlers on its behalf, and ChatGPT-User, the one that opens a page when someone pastes a link, "is not used for crawling the web in an automatic fashion".
What is robots.txt? It is a plain text file at the root of a site that tells crawlers which paths they may fetch, and crawlers read it by the rules of RFC 9309. A crawler looks for the group that names its own token and obeys that group. Only "if no matching group exists" does it fall back to the group for User-agent: *. So a site with a User-agent: GPTBot group and a separate User-agent: * group gives GPTBot only the first set of rules, and every bot without its own group gets the second. The RFC also says that "these rules are not a form of access authorization". robots.txt asks, and well-behaved crawlers comply.
The User-Agent name is just text that anyone can copy, so a request that says GPTBot in a server log is not proof that OpenAI sent it. How to tell real AI bots from fakes is covered in verifying AI crawlers.
Which AI crawler feeds which AI product
These AI crawler user agents are the names to look for in robots.txt and in server logs.
| Operator and product | Search bot: allow it to appear in answers | Training control | User-triggered fetcher and robots.txt |
|---|---|---|---|
| OpenAI: ChatGPT search | OAI-SearchBot | GPTBot | ChatGPT-User: rules may not apply |
| Anthropic: Claude | Claude-SearchBot | ClaudeBot | Claude-User: honours robots.txt |
| Google: Search, AI Overviews, AI Mode | Googlebot | Google-Extended (Gemini models and apps, not Search) | User-triggered fetchers: generally ignore robots.txt |
| Microsoft: Bing, Copilot | Bingbot | Shared training group in robots.txt, from February 2027 | Not documented separately |
| Perplexity | PerplexityBot | None: PerplexityBot is not used for training | Perplexity-User: generally ignores robots.txt |
| Apple: Siri, Spotlight, Safari | Applebot | Applebot-Extended | Not documented separately |
| Meta: Meta AI | Meta-WebIndexer | Meta-ExternalAgent | Meta-ExternalFetcher: may bypass robots.txt |
| Amazon: Alexa | Amzn-SearchBot | Amazonbot | Amzn-User: may not follow all rules |
| DuckDuckGo: AI-assisted answers | DuckAssistBot | None: not used for training | Not documented separately |
CCBot is another name found in AI block lists, including the managed robots.txt example in Cloudflare's documentation. It crawls for Common Crawl, which maintains an open repository of web crawl data that anyone can download and analyse. It checks robots.txt before fetching a page.
Why can't ChatGPT see my website? The technical reasons
When an AI assistant ignores a site or cannot open a page, the technical cause is usually on a short list, and it is the same for Claude, Perplexity, Copilot and the rest.
robots.txt blocks the wrong bot
One mistake is blocking by brand. A site adds User-agent: GPTBot with Disallow: / to keep its content out of training, which is exactly what that does, and ChatGPT search is unaffected. The opposite happens with a long block list copied from the internet: it names every AI bot, OAI-SearchBot, Claude-SearchBot and PerplexityBot included, and the site disappears from AI answers along with training.
Two other patterns cause the same loss. A User-agent: * group with Disallow: / closes the site to every bot that has no group of its own in that file, so writing robots.txt for AI crawlers means naming the search bots you want to keep. And blocking Googlebot or Bingbot takes the site out of AI Overviews and Copilot along with ordinary search.
Cloudflare or another CDN blocks AI crawlers
robots.txt can look wide open while the network in front of the site turns bots away. Does Cloudflare block AI crawlers by default? Since July 1, 2025 it asks every new domain at sign-up whether to allow AI crawlers, and its "AI Crawler and Agent access policies" now sort traffic into Search, Agent and Training, each with its own setting. Since September 2026 new domains also see recommended settings based on how the site makes money, and for ad-supported sites those include disallowing training and blocking agents on pages that carry ads.
The setting that catches people out is Block for training. Since September 15, 2026 it also applies to mixed-use crawlers, the ones that serve search and training at once, and Cloudflare is explicit about the result: "It will stop Applebot, Bingbot, and Googlebot from reaching your site — search included." A site owner who wanted to keep AI training out can end up missing from Google Search, AI Overviews and Copilot. To stop training and keep search, Cloudflare's own advice is the Disallow AI Training setting, named for the Disallow line it publishes in robots.txt, which leaves search crawlers alone.
Cloudflare blocking AI crawlers is not the only edge case. A challenge page for unknown clients, a firewall rule against data-centre addresses or a rate limit tuned for people stops a crawler just as surely. The bot gets a 403 or an empty page, and the robots.txt that allows it never comes into play.
The text only appears after JavaScript runs
Googlebot renders pages: Google Search "runs JavaScript with an evergreen version of Chromium". Apple renders too, and warns that if scripts and styles are blocked in robots.txt, Applebot "may not be able to render the content properly". OpenAI, Anthropic and Perplexity do not say in their crawler documentation whether their bots run JavaScript at all. The safe assumption for a site owner is that a crawler may read only the HTML the server sends. A page whose text, prices or product details are built in the browser by scripts can reach such a crawler as an empty shell.
noindex, nosnippet and other page rules
Page-level rules matter as much as robots.txt for some operators. For Google, a page must be indexed and allowed a snippet to appear as a supporting link, and nosnippet, data-nosnippet, max-snippet and noindex limit what Google shows, AI features included. Apple says nosnippet opts content out of the broad world-knowledge answers its AI models generate. Microsoft's ai-grounding:disallow keeps a page out of Copilot's answers from 2027. OpenAI, Anthropic and Perplexity name robots.txt as their control and say nothing about noindex.
Logins, paywalls and blocked countries
A crawler cannot sign in. Pages behind a login, a paywall or a cookie wall that hides the content until someone clicks are invisible to every bot on this list. A site that blocks traffic by country can also block crawlers whose servers sit in a blocked region.
When ChatGPT says it cannot open a link
Can ChatGPT access websites? Yes, when the site lets its fetcher through. When someone pastes a link and the assistant cannot open it, the fetch came from ChatGPT-User, Claude-User or Perplexity-User and met one of the blocks above. The fetchers also differ on robots.txt. OpenAI says robots.txt "may not apply" to ChatGPT-User, while Anthropic applies robots.txt to Claude-User too. The same page can open in one assistant and be refused in another.
Should you block AI crawlers? Training vs search
Training and search are separate decisions. A site may not want its writing or code to train models it gets nothing from, and still want to be cited when an AI assistant answers a question. Since most operators split training from search, a site can say no to one and yes to the other.
To block AI crawlers for training while staying in AI search, robots.txt disallows the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot) and leaves the search bots alone (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot). Google says Google-Extended is not a ranking signal, and OpenAI and Anthropic treat training and search as separate settings.
A shared signal is arriving to make this simpler. The training preference signal from the IETF's AIPREF work uses a single robots.txt group, User-agent: __usage_ai-training, that tells every participating operator not to train on the site. Anthropic has recognised the signal since October 30, 2026, Microsoft starts on February 1, 2027, and Google says it intends to support it in early 2027. A rule that names ClaudeBot or Google-Extended directly still takes priority for those operators.
Do AI crawlers respect robots.txt? The major operators document robots.txt controls for their training and search crawlers, and Google, Anthropic, Apple and Amazon state outright that they obey them. The user-triggered fetchers are where it varies, as the table shows. And since robots.txt only asks, a site that needs a hard stop has to enforce it at the server or the CDN, with the trade-offs described above.
Browser agents: AI that visits like a person
Browser agents are a fourth kind of visitor. Agents such as ChatGPT's cloud browser, Anthropic's Claude-Agent and Google-Agent run a real browser on the operator's servers and work through a site to finish a task for a user. Anthropic adds the token Claude-Agent to the browser's User-Agent and says preferences for its agent are separate from those for its crawlers. Google lists Google-Agent among its user-triggered fetchers, which generally ignore robots.txt.
Since the User-Agent says little, these agents are starting to sign their requests. OpenAI says "ChatGPT uses Web Bot Auth to identify certain automated browser requests", using HTTP Message Signatures from RFC 9421, and that this applies to its cloud browser, "not web crawlers". Google says it is experimenting with the same protocol for Google-Agent. A signature lets a site check that a request really comes from the operator it names, which is how Web Bot Auth signatures work. For a site owner, the Agent setting at a CDN decides whether users can send an AI agent to the site, separately from search and training.
Does llms.txt help AI crawlers?
llms.txt is a Markdown file at the root of a site that lists its most useful pages for language models. Jeremy Howard proposed it on September 3, 2024. It is often sold as a way into AI search, but the operators say otherwise. Google says sites do not need "machine readable files, AI text files, markup, or Markdown" for its AI features and that "Google Search ignores them". None of the other operators in this article says it reads llms.txt for search.
So does llms.txt work? Not for search visibility. It can still give AI tools that a developer points at the site a clean list of pages, which is what the proposal was written for. Leaving it out costs nothing in search, and publishing one does no harm.
How to check what AI crawlers get
Reading robots.txt by eye misses most of the problems above: the group that only GPTBot reads, the edge block that robots.txt cannot show, the page that is empty without JavaScript. The Agent Readiness Checker fetches a site the way a crawler does, without running JavaScript, and reports bot by bot which AI search bots, training crawlers, assistants and search engines robots.txt lets in, what the home page contains without JavaScript, whether noindex or snippet rules hide it, whether bot protection answered instead of the site, and the files a site publishes for agents. It gives two scores and a list of fixes, the costliest first.
When a bot that claims to be GPTBot or ClaudeBot shows up in the server log, the Crawler Verifier checks the address against what the operator publishes and tells a real crawler from a copy.
FAQ
Should I block AI crawlers on Cloudflare?
Block training if you want, but use Disallow AI Training for it rather than Block, because since September 2026 Block also stops Googlebot, Bingbot and Applebot, search included. The Search and Agent settings decide whether AI search bots and AI agents reach the site at all.
What search engine does Claude use?
Anthropic does not name the search provider behind Claude's web search. It documents Claude-SearchBot, which crawls to improve Claude's search results, and allowing it in robots.txt is the control Anthropic gives site owners.
Do AI crawlers render JavaScript?
Googlebot does, with an up-to-date version of Chromium, and Apple says Applebot renders pages too. OpenAI, Anthropic and Perplexity do not document JavaScript rendering for their crawlers. Put the main text of a page in the HTML the server sends, so every crawler gets it whether it runs scripts or not.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler, while ChatGPT search relies on OAI-SearchBot, and OpenAI treats the two settings as independent.
Can I keep AI agents out but stay in AI search?
Yes, at the network level. robots.txt has no established rule for agents, and Cloudflare notes there is not yet "a well-established directive for expressing Disallow preferences to agents". CDNs that sort AI traffic into Search, Agent and Training let a site block the Agent category while Search stays allowed.
Sources
- Overview of OpenAI Crawlers (OpenAI)
- AI features and your website (Google Search Central)
- RFC 9309: Robots Exclusion Protocol (IETF)
- AI Crawler and Agent access policies (Cloudflare)
