Question

Should I allow Claude-SearchBot?

Updated June 7, 2026

If AI answer visibility for Claude experiences is your goal, allowing Claude-SearchBot is usually the practical default.

Keep policy roles separate

• Claude-SearchBot: search-oriented discovery.
• ClaudeBot: training-policy crawler bucket.
• Claude-User: user-triggered retrieval behavior.

Decision scenarios

• Documentation or SaaS site: usually allow Claude-SearchBot on docs, pricing, changelog, and policy pages.
• Sensitive private portal: allow only public explainers; keep account and internal paths blocked.
• Early-stage testing: allow for a limited path group, then expand after log validation.

Quick decision matrix

• You need answer-surface visibility: allow Claude-SearchBot on public pages.
• You only fear training crawl: keep ClaudeBot separate and blocked if needed.
• You handle private data: allow only public docs/guides; do not expose account or API paths.

How to know the policy is working

1. Claude-SearchBot appears in logs on /robots.txt and core public pages.
2. Status mix is mostly 200/304, not persistent 403.
3. No unintended access on private routes.

If your logs stay empty for multiple days, use this runbook before changing policy lines: Why Claude-SearchBot has no hits.

Verification commands

# Top paths requested by Claude-SearchBot
grep -i 'Claude-SearchBot' /var/lib/caddy/logs/llmsfile-access.log | awk '{print $7}' | sort | uniq -c | sort -nr | head -n 20

# Status mix
grep -i 'Claude-SearchBot' /var/lib/caddy/logs/llmsfile-access.log | awk '{print $9}' | sort | uniq -c | sort -nr

# Core crawl-entry status
for p in /robots.txt /sitemap.xml /llms.txt; do curl -s -o /dev/null -w "%{http_code} %url_effective\n" "https://yourdomain.com$p"; done

Set each policy line explicitly, then confirm with logs instead of assuming wildcard rules did the right thing.

If Cloudflare edge security is in front of origin, also review Bot Fight Mode and challenge policy impact.

Use Claude policy template