PreferiumAI CrawlersTechnical SEO
robots.txt and llms.txt solve different problems
Crawler preferences, documentation indexes and agent manifests have different purposes. Check who consumes each file before treating it as an AI-visibility requirement.
Jump to a section
A file can be useful to an agent without being a ranking signal. Before adding an “AI-ready” file to a client site, identify its purpose, which tools consume it and how you will keep it accurate.
robots.txt expresses crawler preferences
The Robots Exclusion Protocol defines a way to tell cooperating crawlers which paths they may fetch. It is not authentication, and it does not force every bot to comply.
Read each provider’s documentation when separating search, training and user-requested access. OpenAI’s bot documentation, for example, distinguishes several purposes and user agents. A rule for one should not be described as a rule for all of them.
A blocked fetch also does not prove that an assistant can never mention the business. It may have other sources. What the rule directly addresses is access by the crawler to which it applies.
llms.txt is a guide to content
The llms.txt proposal describes a concise Markdown file with context and links to useful material. Its August 2026 revision reflects use on documentation sites and by tools that help agents find references.
That is a different job from robots.txt. A documentation index helps a consumer navigate content; it does not enforce permissions or guarantee inclusion in an answer.
Our earlier article said only robots.txt had a documented effect and treated the other files as cost-free insurance. That was too broad. The relevant distinction is between a documented use by a particular tool and evidence of an effect on search citations.
Check what a manifest name means
Names such as agents.txt and agent.json do not, by themselves, identify one universally supported web standard. Ask which specification an integration expects, which version it implements and which fields it actually reads.
Do not assume that publishing a list of permitted actions makes transactions authorized. Authentication, account permissions and confirmation still belong in the system performing the action.
Keep optional files useful and maintained
For a documentation site, a generated llms.txt index may be a useful way to expose the important guides. Check that links work, the material is current and the file reveals nothing that should remain private.
For a general business website, first make the main pages accurate and accessible. Add optional formats when there is a concrete consumer or a justified experiment. Measure the result of that experiment without promising a citation gain simply because the file exists.
