OpenAI crawler guide

OAI-SearchBot vs GPTBot vs ChatGPT-User

These tokens represent different behaviors. Most policy mistakes happen when teams treat them as one crawler and apply a single allow/block rule.

Role split in plain language

OAI-SearchBot: discovery for search-style experiences.
GPTBot: model-improvement crawler bucket.
ChatGPT-User: user-triggered retrieval behavior.

Recommended visibility-first policy

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/links-map.txt

This pattern keeps search visibility open while separating training-policy choices.

Decision matrix

You want AI search traffic now: allow OAI-SearchBot on public docs and policy pages.
Training usage needs review: keep GPTBot blocked until internal sign-off.
Private app routes exist: block account/admin/API private paths regardless of bot choice.

Verification commands after policy updates

# Nginx
grep -Ei "OAI-SearchBot|GPTBot|ChatGPT-User" /var/log/nginx/access.log | tail -n 80
grep -Ei "OAI-SearchBot|GPTBot|ChatGPT-User" /var/log/nginx/access.log | awk '{print $9, $7}' | head -n 80

# Caddy JSON
jq -r 'select(((.request.headers."User-Agent"[0]) // "") | test("OAI-SearchBot|GPTBot|ChatGPT-User"; "i")) | "\(.status)\t\(.request.uri)\t\((.request.headers."User-Agent"[0]) // "")"' /var/lib/caddy/logs/llmsfile-access.log | head -n 80

Common misreads

Seeing ChatGPT-User and assuming search crawler access is configured.
Allowing all bots globally and forgetting private path blocks.
Updating robots.txt but forgetting Cloudflare/WAF edge rules still challenge requests.

Read ChatGPT-User policy notes Test one robots rule

OpenAI crawler documentation