Question

How to test a robots.txt rule

When your crawl policy looks correct but behavior is unexpected, test one user-agent and one path instead of scanning the whole file manually.

Fast method

1. Copy the live robots.txt from your canonical domain.
2. Enter a user-agent token, such as OAI-SearchBot or GPTBot.
3. Test the exact path you care about, like /pricing or /private/guide.
4. Check which rule wins (longest matching rule should decide).

Open robots.txt rule tester