Question

How to check OAI-SearchBot visits

Updated June 7, 2026

If you changed robots or edge rules, validate with raw logs first. Repeated blind edits usually make diagnosis slower.

Fast checks

grep -i "OAI-SearchBot" /var/log/nginx/access.log | tail -n 50
grep -i "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -nr | head

Caddy JSON log checks

jq -r 'select(((.request.headers."User-Agent"[0] // "") | test("OAI-SearchBot"; "i"))) | [.ts, .request.uri, .status] | @tsv' /var/lib/caddy/logs/llmsfile-access.log | tail -n 120
jq -r 'select(((.request.headers."User-Agent"[0] // "") | test("OAI-SearchBot"; "i"))) | (.request.uri // "/")' /var/lib/caddy/logs/llmsfile-access.log | sort | uniq -c | sort -nr | head -n 20
jq -r 'select(((.request.headers."User-Agent"[0] // "") | test("OAI-SearchBot"; "i"))) | (.status // 0)' /var/lib/caddy/logs/llmsfile-access.log | sort | uniq -c | sort -nr

What to verify

1. Requests hit /robots.txt and key content paths.
2. Status codes are mostly 200 or expected redirects.
3. Your wildcard rules are not blocking by accident.

Example readings from a real access log

• Healthy discovery: OAI-SearchBot requests /robots.txt, /sitemap.xml, then two or more content pages with 200 or 304.
• Policy-only crawl: the bot checks /robots.txt repeatedly but never reaches content pages. Check sitemap URLs, internal links, and whether important pages are hidden behind scripts.
• Edge block: requests reach content URLs but return 403. Check Cloudflare security events before changing robots.txt.
• Wrong token search: logs show GPTBot or ChatGPT-User, but no OAI-SearchBot. Report them separately; they serve different purposes.

Zero-hit diagnosis sequence

1. Confirm token accuracy: search for OAI-SearchBot, not generic OpenAI.
2. Check both apex and www robots responses for policy drift.
3. Inspect Cloudflare security events for challenge/managed-rule blocks.
4. Verify crawl-entry URLs return stable 200/301 from outside your origin network.

curl -s https://yourdomain.com/robots.txt | sed -n '1,80p'
curl -s https://www.yourdomain.com/robots.txt | sed -n '1,80p'
for p in /robots.txt /sitemap.xml /sitemap-index.xml /llms.txt; do
  curl -s -A "OAI-SearchBot/1.0" -o /dev/null -w "%{http_code} %{url_effective}\n" "https://yourdomain.com$p"
done

Avoid false conclusions

1. Separate OAI-SearchBot from GPTBot and ChatGPT-User in reporting.
2. Compare a 7-day window, not one burst hour.
3. Check whether hits stay on /robots.txt only or expand to content pages.

If you need a repeatable command workflow, use the AI crawler log command builder and then validate the result with the log checker.

If you see repeated 403 on crawl-entry paths, continue with OAI-SearchBot 403 behind Cloudflare. If you see persistent zero hits, run the zero-hit checklist (same edge/policy logic).

Open AI crawler log checker