Crawler reference

AI crawler user agents

AI visibility is no longer one crawler setting. Some bots support search results, some support user-triggered retrieval, and some collect data for model training. Treat these choices separately in robots.txt.

Common user-agent tokens

OAI-SearchBot: OpenAI search discovery for ChatGPT search features.
GPTBot: OpenAI crawler for content that may be used to improve foundation models.
ChatGPT-User: OpenAI user-triggered retrieval agent.
ClaudeBot: Anthropic model-development crawler.
Claude-User: Anthropic user-directed retrieval agent.
Claude-SearchBot: Anthropic search-result quality crawler.
PerplexityBot: Perplexity crawler for its search index.

Visibility-first robots.txt starter

User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/links-map.txt

Decide separately whether to allow training crawlers such as GPTBot or ClaudeBot. Search visibility and model-training permission are different policy questions.