How do I allow OAI-SearchBot in Cloudflare without weakening security?

Keep global bot protection in place, then add a narrowly scoped exception for OAI-SearchBot on public crawl paths only. Exclude admin, login, private, API, checkout, and account routes, and verify results in Security Events and origin logs.

Should I allow OAI-SearchBot by user agent only?

No. A user-agent-only allow rule is too broad because user agents can be spoofed. Combine the crawler condition with public path constraints, managed bot signals where available, rate limits, and post-change log review.

Question

How to allow OAI-SearchBot in Cloudflare WAF

Updated June 8, 2026

You do not need to disable Cloudflare security globally to make ChatGPT search discovery work. The safer approach is to keep strict protection in place, then add a narrow exception for OAI-SearchBot on public crawl paths only.

When this rule is needed

Add a Cloudflare exception only after you see evidence that public pages are blocked. Typical signals are 403 responses for OAI-SearchBot, Security Events showing managed challenges for the crawler, or origin logs showing no successful fetches even though robots.txt allows the bot.

Safe rule strategy

Keep your default managed challenge or bot score policy.
Add an exception only for the crawler token and public crawl paths.
Do not allow admin or private paths in the exception.
Use Cloudflare managed bot signals where available instead of trusting user-agent text alone.
Review Security Events and origin access logs after every change.

Cloudflare expression pattern (example)

This example is intentionally path-limited. Adjust the public paths to match your site structure.

(http.user_agent contains "OAI-SearchBot")
and (http.request.uri.path matches "^/(robots\\.txt|sitemap\\.xml|llms\\.txt|guides/|questions/|tools/)")
and not (http.request.uri.path matches "^/(admin|wp-admin|login|api/private)")

Stronger pattern if verified bot fields are available

If your Cloudflare plan exposes verified bot or bot management fields, prefer them. The exact expression depends on the account features, but the logic should be:

verified_bot_is_true
and user_agent_or_bot_name_matches_oai_searchbot
and path_is_public_content
and path_is_not_sensitive

If you only have user-agent text available, keep the exception narrow and monitor it. A user agent can be spoofed; a user-agent-only allow rule across the whole site is not safe.

Paths that should not be allowed

/admin, /wp-admin, /login, /account
/cart, /checkout, payment routes, and customer portals
private APIs, staging previews, exports, logs, and upload directories
any URL that returns personal, paid, unpublished, or tenant-specific content

Validation checklist

Security Events should show fewer 403 events for OAI-SearchBot paths.
Server logs should show 200/304 on allowed pages.
No growth in exploit probes from the same policy change window.
Public pages still return normal status for Googlebot, Bingbot, and human visitors.

Origin log check

grep -Ei 'OAI-SearchBot|ChatGPT-User' /var/log/nginx/access.log \
  | awk '{print $1, $7, $9, $0}' \
  | tail -50

Look for successful requests on pages you want surfaced in ChatGPT search. If the log still shows 403, check Cloudflare Security Events first. If Cloudflare shows allowed traffic but the origin still blocks it, inspect Nginx, app middleware, or IP allowlists.

Rollback rule

Keep the change reversible. If you see exploit probes hitting allowed paths, unexpected admin hits, or a spike in blocked traffic with spoofed crawler names, disable the rule and narrow the path scope before trying again.

Common mistake

Teams often allow by user-agent only and accidentally open sensitive routes. Always combine the crawler condition with path constraints, and treat the first 24 hours after the change as a monitoring window.

If you still see blocked traffic, continue with why OAI-SearchBot gets 403 behind Cloudflare.

Check crawler behavior in logs