How do I test llms.txt?

Confirm the file is available at /llms.txt, returns HTTP 200, contains stable canonical URLs, avoids thin or private pages, and does not point crawlers to URLs blocked by robots.txt or CDN security rules.

What is the most common llms.txt testing mistake?

The most common mistake is testing only that the file exists. You also need to test every listed URL, canonical behavior, robots.txt policy, redirects, and whether AI crawlers can fetch the pages in server logs.

Question

How to test llms.txt

Updated June 8, 2026

Test llms.txt as a source map, not just as a text file. A useful test checks whether the file is reachable, whether its links are canonical and crawlable, and whether those pages are good enough to quote in AI or search answers.

Fast pass: file availability

The file should live at the domain root and return a clean 200 without login, WAF challenge, or redirect chain.

curl -I https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -40

Good signs: 200, readable text, stable cache headers, and links that point to public canonical pages. Bad signs: 403, 404, JavaScript challenge pages, or a file that only works on one host but not the canonical host.

Link checks

Extract every URL from the file and test it. Do not include URLs that redirect repeatedly, return errors, require JavaScript, or represent duplicate tracking variants.

curl -s https://example.com/llms.txt \
  | grep -Eo 'https?://[^ )]+' \
  | sort -u \
  | while read url; do
      printf '%s ' "$url"
      curl -s -o /dev/null -w '%{http_code} %{url_effective}\n' "$url"
    done

Canonical and duplicate checks

Each listed URL should represent one stable source page. If the page canonicalizes elsewhere, list the canonical version instead.

Prefer final canonical URLs over redirecting URLs.
Avoid campaign parameters, sort parameters, previews, tag archives, and search result pages.
Use the same host pattern as the rest of the site, such as apex only or www only.

Robots and CDN policy checks

A listed page is weak if the relevant crawlers cannot fetch it. Check robots.txt and the CDN layer together.

curl -s https://example.com/robots.txt | grep -Ei 'OAI-SearchBot|GPTBot|Claude|Perplexity|User-agent'

# Server log check after publication
grep -Ei 'OAI-SearchBot|GPTBot|ClaudeBot|PerplexityBot|ChatGPT-User' /var/log/nginx/access.log \
  | awk '{print $7, $9, $0}' \
  | tail -50

Content quality review

Do not list every page. List pages that help a model or search crawler understand the site. Strong entries usually include one clear topic, visible text, recent facts where dates matter, and direct answers. Weak entries include thin category pages, duplicate location pages, pages with no stable content, or pages that are primarily ads.

Release checklist

File returns 200 at /llms.txt.
Every listed URL returns 200 or a single intentional redirect to the canonical URL.
No private, admin, staging, search, tag, preview, or tracking URLs are included.
Listed pages are not blocked by robots.txt or CDN security rules.
The file is linked from your sitemap, footer, docs, or AI policy page if it is part of your public workflow.

Maintenance cadence

Re-check after major content updates, URL migrations, robots policy changes, CDN/WAF changes, and template changes that affect canonical tags. For stable sites, a monthly review is enough; for active documentation or product sites, use a weekly check.

Run validator now