Free browser tool

robots.txt rule tester

Test one crawler decision at a time. Paste a robots file, choose a user-agent token, and enter a URL path to see whether it is allowed or blocked.

robots.txt User-agent: * Allow: / Disallow: /private/ User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml User-agent token Path to test

What this helps you debug

Use it when a path is unexpectedly blocked, when wildcard rules overlap, or when you need to verify that AI search crawlers and training crawlers have different policies.