Unknown error
Malformed origin responses, oversized headers, or abrupt resets.
Open 520 guideTroubleshooting hub
Cloudflare AI crawler troubleshooting hub
Use this hub when AI crawlers cannot reach your site reliably. Start with the matching failure pattern, then move to logs and rule tuning.
Step 1: Identify the failure class
- 5xx error page shown by Cloudflare: start with Cloudflare 520-526 differences.
- Unsure whether the issue is malformed response or timeout: use 520 vs 524 quick split.
robots.txtallows crawl but logs show 403: start with OAI-SearchBot 403 behind Cloudflare.- Zero
Claude-SearchBothits in 72h: use Claude-SearchBot zero-hit checklist. - Need safe allow rules without opening sensitive paths: use allow OAI-SearchBot in Cloudflare WAF.
Example: choose by log symptom
If your logs show repeated Googlebot, OAI-SearchBot, or ClaudeBot requests with 200/304 on public pages, the crawler path is probably open. The next question is indexing, internal links, and source-page quality; use the Search Console submission workflow when deciding which source pages to inspect first. For Gemini policy decisions, separate crawl access from the Google-Extended robots token, then use the Google-Extended allow/block decision guide before changing rules.
If your logs show crawler requests only for /robots.txt and never for your question, guide, or product pages, check sitemap freshness, whether your important URLs are linked from crawlable HTML, and whether your llms.txt link list is focused enough. If requests appear but return 403, start with WAF events before editing robots.txt.
If most noise is scanner paths such as /.env, /wp-login.php, or phpunit, do not let those hits drive content work. We document this filtering approach in the LLMs File traffic baseline case study.
Step 2: 520-526 fix pages
Web server down
Connection refused at origin process or firewall layer.
Open 521 guideConnection timed out
Origin latency or path-level timeout bottlenecks.
Open 522 guideOrigin unreachable
DNS-to-origin mismatch or route availability issues.
Open 523 guideTimeout occurred
Long-running application responses behind Cloudflare limits.
Open 524 guideTLS and certificate failures
Handshake and certificate-chain mismatch between edge and origin. Use the Caddy 525 guide when Caddy owns origin TLS.
Open Nginx 525 pathStep 3: Verify crawler access after fixes
- Check security events for reduced blocks on public crawl paths.
- Check origin logs for crawler status mix (200/304 should dominate).
- Track
/robots.txt,/sitemap.xml, and key question pages for stable access over multiple days.
Use these pages as your post-fix verification path: how to check OAI-SearchBot visits, how to check ChatGPT-User visits, and PerplexityBot rollout decision guide.