Question
How to block AI crawlers
Block AI crawlers by adding Disallow rules for their robots.txt tokens, and enforce at the edge (WAF/firewall) for any that ignore robots.txt. Decide per purpose — blocking training bots while keeping search bots preserves your AI answer visibility.
Block training, keep search visibility
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Keep these so you stay citable:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Block everything (rarely advisable)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: CCBot
Disallow: /Enforce at the edge if ignored
For crawlers that do not honor robots.txt, add a WAF/firewall rule matching the user-agent and verified IP ranges. robots.txt alone will not stop them.
Related pages: do AI crawlers respect robots.txt, should I allow GPTBot, the GEO checklist.