From 2023 onward, many sites added AI bots to their robots.txt file to stop their content from being used for model training. The intent was reasonable, but in practice a critical distinction was missed: the same company’s training bot and its search/response bot are different agents. Block both and you don’t just drop out of the training data — you also destroy your chance of being cited in responses.
Two different kinds of bot
| Kind | What it does | If you block it |
|---|---|---|
| Training bot | Collects content to use in model training | Visibility isn’t directly affected |
| Search / response bot | Fetches current content when a user asks a question | You don’t get cited in responses |
The bots that are critical for visibility
- OAI-SearchBot — needed for ChatGPT to be able to show your site in its search results.
- ChatGPT-User — reads your page when a user opens a link.
- PerplexityBot and Perplexity-User — needed to be cited in Perplexity responses.
- ClaudeBot, Claude-User, Claude-SearchBot — crawling and fetching on the Claude side.
- Googlebot — the core crawler for both classic search and AI Overviews.
- Bingbot — most of Copilot’s responses are fed from the Bing index.
By contrast, GPTBot (OpenAI training), Google-Extended (Gemini training) and Applebot-Extended (Apple Intelligence training) are not visibility bots. Blocking them does not, on its own, stop you from being mentioned in responses.
A practical rule: if you don’t want your content used for model training but do want to be mentioned in responses, block the training bots and allow the search/response bots.
An example configuration
# Training bots blocked
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Search / response bots allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
Sitemap: https://example.com/sitemap.xmlrobots.txt isn’t the only blocker
- CDN and WAF rules (Cloudflare bot management, for instance) can block AI bots independently of robots.txt.
- Server-side rules that return a 403 based on the user agent quietly make you invisible.
- Content generated only with JavaScript can’t be read by some bots.
- Content behind a login wall can never be cited.
That is why the audit shouldn’t stop at robots.txt: the CDN rules, the server logs and the server-side rendering state have to be examined alongside it.