Crawler reference
AI crawler user agents
AI visibility is no longer one crawler setting. Some bots support search results, some support user-triggered retrieval, and some collect data for model training. Treat these choices separately in robots.txt.
Common user-agent tokens
OAI-SearchBot: OpenAI search discovery for ChatGPT search features.GPTBot: OpenAI crawler for content that may be used to improve foundation models.ChatGPT-User: OpenAI user-triggered retrieval agent.ClaudeBot: Anthropic model-development crawler.Claude-User: Anthropic user-directed retrieval agent.Claude-SearchBot: Anthropic search-result quality crawler.PerplexityBot: Perplexity crawler for its search index.
Visibility-first robots.txt starter
User-agent: *
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/links-map.txtDecide separately whether to allow training crawlers such as GPTBot or ClaudeBot. Search visibility and model-training permission are different policy questions. For Google's Gemini-specific policy token, use the Google-Extended explanation before editing robots.txt.
Verification pages to run after policy updates
- How to check OAI-SearchBot visits
- How to check ChatGPT-User visits
- How to check GPTBot visits
- How to check Claude-SearchBot visits
- How to check PerplexityBot visits
If robots rules are correct but crawlers still see 403, continue with Cloudflare Bot Fight Mode troubleshooting.