Question

How to check ClaudeBot visits

Updated June 7, 2026

Use the exact token ClaudeBot. Do not merge it with Claude-SearchBot in one count.

Quick commands

grep -i "ClaudeBot" /var/log/nginx/access.log | tail -n 80
grep -i "ClaudeBot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -nr | head
grep -i "ClaudeBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -nr

Caddy JSON log commands

jq -r 'select(((.request.headers."User-Agent"[0] // "") | test("ClaudeBot"; "i"))) | [.ts, .request.uri, .status] | @tsv' /var/lib/caddy/logs/llmsfile-access.log | tail -n 120
jq -r 'select(((.request.headers."User-Agent"[0] // "") | test("ClaudeBot"; "i"))) | (.request.uri // "/")' /var/lib/caddy/logs/llmsfile-access.log | sort | uniq -c | sort -nr | head -n 20
jq -r 'select(((.request.headers."User-Agent"[0] // "") | test("ClaudeBot"; "i"))) | (.status // 0)' /var/lib/caddy/logs/llmsfile-access.log | sort | uniq -c | sort -nr

What to look for

• High 200 ratio on core pages means crawler access is clean.
• Frequent 403 suggests policy or firewall blocking.
• Mostly /robots.txt hits means discovery started but deep crawl is still light.

Sanity checks that prevent false conclusions

1. Separate ClaudeBot, Claude-SearchBot, and Claude-User into different counters.
2. Compare today and 7-day windows before changing policy.
3. Check /robots.txt, /sitemap.xml, and top public pages first.

If you see repeated 403 on crawl-entry paths, continue with Cloudflare Bot Fight Mode crawler checks.

Generate commands for your stack