Question

What is ClaudeBot?

Updated June 22, 2026

ClaudeBot is Anthropic's web crawler. Anthropic documents three separate agents: ClaudeBot (general crawling/training), Claude-SearchBot (search indexing), and Claude-User (live fetch when a user asks). Treat them separately in robots.txt.

The three Claude agents

• ClaudeBot: general/ training crawl.
• Claude-SearchBot: indexes pages for Claude search results.
• Claude-User: fetches a page in real time for a user request.

Visibility-first policy

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

Related pages: should I allow Claude-SearchBot, check ClaudeBot visits, Claude robots.txt guide.

Build your AI robots.txt