Question

Why is GPTBot still blocked?

Updated June 7, 2026

Most cases come from policy conflicts, not from a single missing line.

Fast troubleshooting list

1. Check for broad rules like User-agent: * with global Disallow.
2. Confirm no CDN feature is replacing your robots.txt file.
3. Verify the live file over HTTPS on the canonical domain.
4. Check logs to confirm requests and status codes after the change.

Timing expectation

Policy updates are not always reflected instantly in crawler behavior. Keep the file stable long enough to observe fresh crawl attempts in logs.

Audit crawler logs