Unknown error
Malformed origin responses, oversized headers, or abrupt resets.
Open 520 guideTroubleshooting hub
Cloudflare AI crawler troubleshooting hub
Use this hub when AI crawlers cannot reach your site reliably. Start with the matching failure pattern, then move to logs and rule tuning.
Step 1: Identify the failure class
- 5xx error page shown by Cloudflare: start with Cloudflare 520-526 differences.
- Unsure whether the issue is malformed response or timeout: use 520 vs 524 quick split.
robots.txtallows crawl but logs show 403: start with OAI-SearchBot 403 behind Cloudflare.- Zero
Claude-SearchBothits in 72h: use Claude-SearchBot zero-hit checklist. - Need safe allow rules without opening sensitive paths: use allow OAI-SearchBot in Cloudflare WAF.
Step 2: 520-526 fix pages
Web server down
Connection refused at origin process or firewall layer.
Open 521 guideConnection timed out
Origin latency or path-level timeout bottlenecks.
Open 522 guideOrigin unreachable
DNS-to-origin mismatch or route availability issues.
Open 523 guideTimeout occurred
Long-running application responses behind Cloudflare limits.
Open 524 guideTLS and certificate failures
Handshake and certificate-chain mismatch between edge and origin.
Open 525/526 pathStep 3: Verify crawler access after fixes
- Check security events for reduced blocks on public crawl paths.
- Check origin logs for crawler status mix (200/304 should dominate).
- Track
/robots.txt,/sitemap.xml, and key question pages for stable access over multiple days.
Use these two pages as your post-fix verification path: how to check OAI-SearchBot visits and PerplexityBot rollout decision guide.