What is llms.txt?
The llms.txt proposal suggests placing a Markdown file at /llms.txt on the root of a website.
The file can provide a short summary and a curated list of important links. Its stated purpose is to help language models use website information at inference time, when a system is answering a request.
That is a proposal. It is not an internet standard like the Robots Exclusion Protocol. It does not define how a search or answer engine must discover, process, rank or cite the file.
The distinction matters because “the format exists” and “the platforms use it for visibility” are two different claims.
What does Google say?
Google’s current AI features and website guidance says its Search systems do not use llms.txt.
Google also says publishing the file neither helps nor harms visibility or ranking in its systems. The same search foundations still apply: pages need to be accessible, indexable, useful and clear enough to serve the searcher’s need.
That does not prove every tool ignores llms.txt. It answers a narrower and important question: do not sell the file as a Google Search or Google AI inclusion control.
Google can change its systems. Any future claim should be checked against current platform documentation rather than carried forward from this article.
What do AI providers document?
Major providers document named crawlers and access controls.
| Provider | Documented crawler examples | Documented owner control |
|---|---|---|
| OpenAI | OAI-SearchBot, GPTBot, ChatGPT-User | robots.txt rules and access configuration |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User | Separate robots.txt controls by crawler role |
| Perplexity | PerplexityBot, Perplexity-User | robots.txt, published bot details and IP information |
OpenAI’s publisher FAQ identifies OAI-SearchBot as the crawler relevant to ChatGPT search. Anthropic’s crawler documentation and Perplexity’s crawler documentation make similar role distinctions.
Those controls address access. They still do not promise a citation, recommendation or customer action.
llms.txt is not robots.txt
The names look similar. Their jobs are not.
| File or control | Main job | What it cannot prove |
|---|---|---|
robots.txt | Express crawler access preferences | That blocked information is secure |
| Robots meta | Give page-level indexing or serving instructions where supported | That a page will rank |
| XML sitemap | List preferred URLs for discovery | That a search engine will index them |
| Canonical HTML page | Provide the actual source content | That an assistant will cite it |
llms.txt | Offer a curated Markdown map under a voluntary proposal | That a provider will fetch or use it |
| Server log | Record a request to the file or page | That the fetched content changed an answer |
RFC 9309 standardizes the Robots Exclusion Protocol. It also makes clear that robots.txt is not access authorization. llms.txt is further removed: it is a content map, not a crawl permission or security boundary.
What does the current evidence show?
Server-log studies provide better evidence than speculation because they can show whether a file was requested.
Ahrefs analyzed 137,210 domains using its Web Analytics product in May 2026. Its llms.txt study reported that 97% of valid files received no requests during that month. The sample likely skews toward technical and SEO-aware sites, so it should not be treated as the whole web.
OtterlyAI reported a 90-day experiment with more than 62,100 AI bot visits. Its /llms.txt file received 84 of them, about 0.1% of the recorded AI bot requests. That is one site and one period.
The useful conclusion is not “the file never works.” It is this: current request evidence does not justify treating llms.txt as a major visibility lever.
A fetch is not a citation
Visibility claims often skip several steps.
- The file exists.
- A crawler requests it.
- A system processes the content.
- The content influences an answer.
- The answer cites or names the business.
- The right buyer takes a qualified action.
A server log can establish step two. It cannot establish the remaining steps by itself.
A citation is also not a qualified call, appropriate appointment, prospective-client enquiry, sales opportunity or completed order. Measure each stage separately. Otherwise, a cheap file becomes a large claim that the evidence cannot support.
When might a small experiment make sense?
A small experiment may be reasonable when all four conditions are true:
- The site has a deep documentation library that is hard for agents to navigate.
- The file can be generated from canonical content without manual drift.
- Server logs can identify requests to
/llms.txt. - The owner accepts that the test may show no measurable use.
B2B SaaS documentation is the clearest example. A developer portal may have hundreds of technical pages, versions and references. A maintained map could help an agent find the right documentation even if it never improves public AI-search visibility.
For a small service website with ten clear pages, the file is less likely to earn its maintenance cost.
Run the experiment without fooling yourself
Record the test before publishing the file.
| Field | Decision |
|---|---|
| Hypothesis | Which named crawler or agent is expected to request the file? |
| Version | What canonical links and summary does the file contain? |
| Access | Are documented crawlers allowed to reach the file and linked pages? |
| Log window | How long will requests be observed? |
| Answer sample | Which fixed questions will be checked separately? |
| Stop rule | When will the file be removed or left unmaintained? |
Do not change crawler rules, content architecture and llms.txt at the same time if the goal is to learn which change mattered.
At the end of the window, report requests, sampled answer observations and qualified business actions as separate lines.
The decision changes by industry
The evidence standard stays the same. The likely value and maintenance risk change.
| Industry | Recommended starting position | Business outcome kept separate |
|---|---|---|
| Home Services | Usually skip; improve service, location and trust pages first | Qualified calls and booked jobs |
| Healthcare | Usually skip; prioritize accurate provider/service information and privacy-safe access | Appropriate appointments |
| Legal | Usually skip; prioritize clear, jurisdiction-reviewed practice information | Qualified consultations and accepted matters |
| B2B SaaS | Consider a cheap test for large, structured documentation libraries | Qualified demos, opportunities and pipeline |
| eCommerce | Use caution; fast-changing catalogs can make a manual file stale | Completed and return-adjusted orders |
Healthcare content remains subject to clinical and privacy review. Legal content remains subject to jurisdiction, advertising and confidentiality review. llms.txt changes none of those duties.
Fix the controls that providers actually document
Before adding another file, check whether important canonical pages are accessible and whether the relevant provider crawlers are intentionally allowed or blocked.
Mindflow’s AI Visibility service examines the public sources, access conditions and repeated answer observations that can be checked. The AI Search Guide explains the wider discovery and answer process without treating one file as the system.
The AI answer variation guide shows why one sampled response cannot establish a stable recommendation or universal result.
If you have a crawler-access question, contact Mindflow with the public URL and a safe excerpt from the server log. Mindflow can help compare the documented controls before you add another file. No change can promise an AI citation or recommendation.
You can also read the Mindflow methodology to see how a test becomes a priority.