Comparison

llms.txt vs robots.txt

robots.txt controls crawler access. llms.txt gives AI assistants a curated map of important pages. They work together, but one cannot replace the other.

Use robots.txt for permissions

Robots.txt tells crawlers which paths they may fetch. For AI visibility, you can allow search and user-retrieval crawlers while deciding separately whether training crawlers fit your content policy.

Use llms.txt for guidance

LLMs.txt should explain the site and list the strongest source pages. It does not grant permission to crawl blocked pages, and it does not force AI systems to cite you.

Practical setup

User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/links-map.txt

Then publish a separate /llms.txt that points to the pages AI assistants should use as sources.