Crawler policy

AI crawler robots.txt builder

Use this starter policy to align your AI crawler preferences with your SEO goals. If your strategy is to earn citations from AI answer engines, blocking every AI crawler can work against that goal.

Practical guidance

Robots.txt is an access policy, not a ranking boost. Use it to express which crawlers may fetch your content. For AI search visibility, allow search and user-retrieval crawlers while deciding separately whether training crawlers fit your policy.

OpenAI documents OAI-SearchBot for search discovery and GPTBot for model improvement. Anthropic documents Claude-SearchBot, Claude-User, and ClaudeBot. Perplexity documents PerplexityBot for its index.

Already have a file? Use the AI robots.txt checker to audit it.