Cloudflare troubleshooting

Why Claude-SearchBot has no hits

If your 72-hour log window shows zero Claude-SearchBot lines, start with evidence, not assumptions. Most incidents are caused by one of four layers below.

Layer 1: Token mismatch in your query

• Claude-SearchBot is not the same as ClaudeBot or Claude-User.
• If your grep pattern is just Claude, results are easy to misread.

grep -i "Claude-SearchBot" /var/log/nginx/access.log | wc -l
grep -i "ClaudeBot" /var/log/nginx/access.log | wc -l
grep -i "Claude-User" /var/log/nginx/access.log | wc -l

Layer 2: Cloudflare edge blocking on public paths

A common pattern is robots policy set to allow, but edge security still returns challenge or 403.

1. Open Cloudflare Security Events and filter on paths like /robots.txt, /sitemap.xml, and key guides.
2. Check managed rules and bot controls that trigger on crawler-like behavior.
3. If needed, add a narrow allow rule for known crawl-entry paths only.

Related runbook: Does Bot Fight Mode block AI crawlers?

Layer 3: Robots policy drift between hosts or deploy versions

It is common to update one host and forget the other, or to keep stale file copies after deploy.

curl -s https://llmsfile.com/robots.txt | sed -n '1,80p'
curl -s https://www.llmsfile.com/robots.txt | sed -n '1,80p'

If content differs, fix host-level consistency first.

Layer 4: Crawl-entry URLs are not consistently reachable

• /sitemap.xml can return 301 to /sitemap-index.xml, which is acceptable if final URL returns 200.
• If final status is non-200 or intermittently failing by region, crawler discovery slows down.

for p in /robots.txt /sitemap.xml /sitemap-index.xml /feed.xml; do
  curl -s -A "Claude-SearchBot/1.0" -o /dev/null -w "%{http_code} %{url_effective}\n" "https://yourdomain.com$p"
done

Decision point after checks

• You find 403/challenges: tune Cloudflare rules for public crawl entries.
• You find mismatched robots: redeploy unified policy files.
• All checks pass but still zero hits: keep monitoring 7-14 days and focus on content freshness and internal links on high-value pages.

Run Claude-SearchBot visit checks