Question

How to check Claude-SearchBot visits

Updated June 7, 2026

Do not treat every Claude* line as the same thing. Keep Claude-SearchBot, ClaudeBot, and Claude-User separate or your decisions will drift.

Fast checks by web server type

# Nginx
grep -i "Claude-SearchBot" /var/log/nginx/access.log | tail -n 30
grep -i "Claude-SearchBot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -nr | head -n 20
grep -i "Claude-SearchBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -nr

# Caddy JSON logs
jq -r 'select(((.request.headers."User-Agent"[0]) // "") | test("Claude-SearchBot"; "i")) | "\(.status)\t\(.request.uri)"' /var/lib/caddy/logs/llmsfile-access.log | head -n 40
jq -r 'select(((.request.headers."User-Agent"[0]) // "") | test("Claude-SearchBot"; "i")) | .status' /var/lib/caddy/logs/llmsfile-access.log | sort | uniq -c | sort -nr

What healthy evidence looks like

• Core crawl-entry files are reachable: /robots.txt, /sitemap.xml or /sitemap-index.xml, and important guide pages.
• Status mix is mostly 200 and 304, not repeated 403/429.
• Hits appear across multiple days, not only a one-time probe.

When you see zero Claude-SearchBot hits

1. Confirm you are checking the right token (Claude-SearchBot, not ClaudeBot).
2. Check Cloudflare security events for challenge/managed-rule blocks on public paths.
3. Verify robots policy on both apex and www hostnames.
4. Check whether crawler entry paths return 200/301 globally and not only from your region.

Use this focused runbook when your logs stay empty: Why Claude-SearchBot has no hits.

Sample daily check block

for p in /robots.txt /sitemap.xml /sitemap-index.xml /questions/how-to-check-claude-searchbot-visits/; do
  curl -s -A "Claude-SearchBot/1.0" -o /dev/null -w "%{http_code} %{url_effective}\n" "https://yourdomain.com$p"
done

Classify crawler lines