Should I block ClaudeBot but allow Claude-SearchBot?

For many visibility-first sites, yes. Allowing Claude-SearchBot and Claude-User keeps search and user-directed retrieval open, while blocking ClaudeBot signals that future content should be excluded from Anthropic model-training crawls.

Does robots.txt need to be configured per subdomain for Claude crawlers?

Yes. Robots.txt is evaluated per host. Configure the apex domain, www host, docs host, app host, or any other subdomain separately when they expose different content.

Can I use Crawl-delay with ClaudeBot?

Anthropic says its bots support the non-standard Crawl-delay extension. Use it when you want to limit crawl pace without fully blocking a bot.

Claude crawler guide

ClaudeBot, Claude-User, and Claude-SearchBot

Updated July 6, 2026. Anthropic documents three different bot tokens. ClaudeBot is connected to model development, Claude-User supports user-directed page retrieval, and Claude-SearchBot supports search response quality.

Recommended visibility-first policy

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

Sitemap: https://example.com/links-map.txt

This keeps Claude search and user retrieval open while opting out of model-development crawling. If your organization wants maximum crawlability and accepts training use, you can allow ClaudeBot too.

Which Claude crawler should you allow?

Goal	Claude-SearchBot	Claude-User	ClaudeBot
Appear accurately in Claude search-style answers	Allow	Allow	Optional
Let a Claude user fetch your page during a conversation	Optional	Allow	Optional
Opt out of future model-training crawls	Allow	Allow	Disallow
Reduce all Anthropic crawler access	Disallow	Disallow	Disallow

The important detail is separation. Blocking ClaudeBot is not the same as blocking Claude-SearchBot or Claude-User. If traffic acquisition is the goal, do not block the retrieval and search bots by accident.

Use Crawl-delay when load is the concern

If the concern is crawl rate rather than policy, Anthropic says its bots support the non-standard Crawl-delay extension. Use it to slow a bot before you block it entirely.

User-agent: ClaudeBot
Crawl-delay: 5

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Do not rely only on IP blocking for policy control. If a bot cannot read /robots.txt, it may not receive your preferred directive cleanly.

Apply the rule per host

Robots rules are host-specific. Configure and test each public host separately, especially when your marketing site, docs, app, and blog live on different subdomains.

curl -I https://example.com/robots.txt
curl -I https://www.example.com/robots.txt
curl -I https://docs.example.com/robots.txt

If the apex domain allows Claude but www blocks it, crawlers may see inconsistent policy. Keep the rule identical unless the subdomain has a deliberate reason to differ.

Verify after deployment

After publishing, test both the file and real crawler log lines. A robots file that looks correct in the CMS can still be hidden by redirects, WAF challenges, or a stale CDN cache.

curl -s https://example.com/robots.txt | sed -n '/Claude/,+4p'
grep -Ei 'ClaudeBot|Claude-User|Claude-SearchBot' /var/log/nginx/access.log | tail

If you use Cloudflare or another WAF, confirm these requests are not challenged. A crawler that receives an interstitial or 403 cannot treat the page as a clean source.

Real-world decision pattern

One common setup for content-heavy teams is: allow Claude-SearchBot and Claude-User for answer visibility and user-triggered retrieval, while blocking ClaudeBot until internal policy review is complete. Re-check this after every robots deployment.

Read: Should I allow Claude-SearchBot? Check Claude-SearchBot visits Run policy check

Anthropic crawler documentation