Allow OAI-SearchBot, block GPTBot
Keep answer visibility while limiting training crawler access.
Open policyPolicy library
AI crawler robots.txt policy templates
Use these policy examples when you need precise crawler decisions and clear separation between search visibility and training crawlers.
Allow Claude-SearchBot, block ClaudeBot
Separate search indexing from model-training crawler policy.
Open policyAllow PerplexityBot safely
Keep Perplexity discovery open with explicit policy lines.
Open policyAllow AI search, block training bots
One compact template for common visibility-first setups.
Open policyHow to verify your policy
Validate rules against sample user-agents and logs.
Open guide