Question

How to check OAI-SearchBot visits

If you changed robots rules and want evidence, start with raw logs before adjusting policy again.

Fast checks

grep -i "OAI-SearchBot" /var/log/nginx/access.log | tail -n 50
grep -i "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -nr | head

What to verify

  1. Requests hit /robots.txt and key content paths.
  2. Status codes are mostly 200 or expected redirects.
  3. Your wildcard rules are not blocking by accident.

On Caddy JSON logs, filter by user-agent token and inspect path + status together. Avoid deciding from one line.

Open AI crawler log checker