Free browser tool

AI crawler log checker

Paste one user-agent per line from access logs. This tool classifies known AI crawlers and gives policy recommendations.

User-agent lines Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot) Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot) Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/) Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

How to use this output

Use the counts to align your robots policy with your goals: search visibility, user-triggered retrieval, or training control.