Question
Does Cloudflare Bot Fight Mode block AI crawlers?
It can. If your bot security layer issues challenges, crawler requests may fail even when robots.txt is permissive.
How to verify quickly
- Check Cloudflare Security Events for service field and action around crawler requests.
- Look for repeated 403 or challenge outcomes on
/robots.txt, sitemap, and core pages. - Cross-check origin logs to confirm requests reached the server.
Triage matrix
- Edge 403 + no origin log line: block happened at Cloudflare edge before origin.
- Edge 403 + origin 403 too: both edge and origin policy might be blocking.
- Mostly 200/304 on crawler paths: Bot Fight Mode is likely not your bottleneck.
Evidence commands (copy-paste)
# Origin log check (Caddy default JSON log from this site setup)
grep -E 'OAI-SearchBot|Claude-SearchBot|PerplexityBot' /var/lib/caddy/logs/llmsfile-access.log | tail -n 120
# Status mix for AI crawler lines
grep -E 'OAI-SearchBot|Claude-SearchBot|PerplexityBot' /var/lib/caddy/logs/llmsfile-access.log | awk '{print $9}' | sort | uniq -c | sort -nr
# Are crawl-entry pages reachable?
for p in /robots.txt /sitemap.xml /llms.txt /questions/; do curl -s -o /dev/null -w "%{http_code} %url_effective\n" "https://yourdomain.com$p"; doneWhat to monitor after changes
- 403 rate for
OAI-SearchBotandClaude-SearchBoton public pages. - Status mix for
/robots.txt,/sitemap.xml, and top question pages. - No accidental drop in protection on admin or private API paths.
Safe allow scope pattern
Do not create a global bypass. If you need a temporary exception, scope it to crawl-entry paths only (/robots.txt, /sitemap*.xml, /llms.txt, selected public guides), then keep admin and API routes under strict protection.
Practical fix path
If policy needs AI-search visibility, tune edge security settings first, then validate crawler access again with logs.
For a path-scoped exception pattern, continue with how to allow OAI-SearchBot in Cloudflare WAF.