Few ideas in SEO have spread as fast as a small text file placed at the root of a website. Plugins now generate it in one click, agencies add it to audit checklists, and some vendors describe it as essential for AI search. The reality is more modest. An llms.txt file can be useful in specific situations, but the evidence that major AI assistants read it when answering everyday questions remains thin.
Knowing what the file does, and what it doesn’t, helps teams decide whether it deserves a place on the roadmap.
Where the Idea Came From
The proposal was published in September 2024 by Jeremy Howard of Answer.AI. Its goal was practical: large language models struggle with cluttered web pages full of navigation, scripts, and ads, so a short Markdown file could point them toward the most useful content in a clean format. A clear view of how llms.txt fits into AI readiness starts with that original purpose, which focused on helping AI tools answer questions about a site on demand rather than on training models or improving rankings. Context limits were part of the motivation too. A model can only process a certain amount of text at once, and a heavy web page can waste much of that space on menus, footers, and code that carry no meaning.
The specification is deliberately simple. The file uses Markdown, sits at the root of the domain, and needs only one required element: a top level heading with the site or project name.
What the File Looks Like
A typical version contains a title, a one line summary, and grouped lists of links with short notes. A small software company might publish something like this:
# Example Analytics
> Web analytics software for small online stores, with dashboards, reports, and a public API.
## Documentation
- [Getting started](https://example.com/docs/start.md): Installation and first report
- [API reference](https://example.com/docs/api.md): Endpoints and authentication
## Product
- [Pricing](https://example.com/pricing.md): Plans and limits
- [Integrations](https://example.com/integrations.md): Supported platforms
## Optional
- [Changelog](https://example.com/changelog.md): Release history
The proposal also suggests offering clean Markdown versions of important pages at the same address with “.md” added. By convention, the section named “Optional” holds links an AI tool can skip when it needs less context.
What the Evidence Shows So Far
Publishing has grown quickly. Website platforms and documentation tools now generate the file automatically, and tens of thousands of sites carry one. Publishing, however, is not the same as being read.
Google has been the clearest voice. Its Search Advocate John Mueller said in 2025 that none of the AI services had announced using these files, adding that server logs show they don’t even request them, and compared the idea to the long ignored keywords meta tag. Google has given no sign of supporting the file in Search or AI Overviews.
OpenAI’s crawler documentation directs site owners to robots.txt and makes no mention of the proposal. Anthropic publishes its own version for its documentation, which shows interest in the format, but hasn’t stated that Claude reads these files on other sites when answering questions. Server log studies published in 2026 found that most of these files received few or no requests from AI bots at all.
Where the file does get used is in developer tools. Coding assistants and documentation tools can read it when a user points them at a specific site, which explains why software documentation remains its strongest use case.
Which Sites Benefit Most
The value of the file depends heavily on the type of site and its audience.
| Type of site | Worth adding? | Reason |
|---|---|---|
| Developer documentation and APIs | Yes | Coding assistants and developers’ AI tools can use it to find the right pages quickly |
| Software and SaaS products | Usually | Low effort, and helpful when technical buyers use AI tools to research the product |
| Publishers and blogs | Optional | Little evidence of benefit for general AI search visibility |
| Local service businesses | Low priority | Business profiles, reviews, and consistent listings matter far more |
| Online stores | Low priority | Product data, structured markup, and crawler access deliver more value |
For most businesses outside software, the file is harmless but unlikely to change how often AI assistants mention them. That makes it a reasonable small task, not a strategy.
The simplest way to judge value for a specific site is to check its own server logs after publishing. Filter requests for the file by user agent over a month or two. Regular visits from AI bots or developer tools suggest it is being used, while silence suggests the effort is better spent elsewhere. This turns a debate based on opinions into a decision based on the site’s own data.
Setting One Up Without Overthinking It
Creating the file takes less than an hour. List the pages that best explain the business: the main product or service pages, pricing, key guides, and contact details. Write one plain sentence describing each, group them under simple headings, and save the result as a Markdown text file at the root of the domain.
Keep it short and accurate. A file that lists outdated pages or discontinued services can spread the same errors it was meant to prevent. Review it whenever the site structure changes, and remove links that redirect or no longer exist.
Automatically generated versions from plugins deserve a quick review too. Some list every page on the site, which defeats the purpose of a curated starting point. SEO Sets reports whether the file exists as part of its readiness checks, but treats it as information only, reflecting the fact that engine adoption hasn’t been proven.
What Matters More for AI Visibility Right Now
Teams with limited time get more from the basics that AI systems demonstrably rely on.
The first is crawler access. A site that blocks AI search bots in robots.txt or at the firewall level can’t be cited, no matter what other files it publishes. The second is content in raw HTML. Many AI bots don’t run JavaScript, so content loaded only through scripts may never be seen. The third is clear answers near the top of key pages, which give assistants a quotable passage. The fourth is valid structured data and consistent business information across the web.
These factors make up most of what any serious AI readiness review examines. SEO Sets scores pages on dimensions like these, including quotable answers, structured data, and whether content appears in the raw HTML an AI bot receives.
Answers to Common Questions About the File
Can the file replace robots.txt?
No. It doesn’t control access at all. Bots that should be allowed or blocked still follow robots.txt and server settings.
Does adding the file improve Google rankings?
There’s no evidence that it does. Google has indicated it doesn’t use the file for Search or AI Overviews.
Is there any risk in adding one?
The risk is small. The main concern is accuracy, since an outdated file can point AI tools to pages that no longer reflect the business.
The Verdict
An llms.txt file is a sensible addition for documentation sites and software products, and a harmless one for most other businesses. It isn’t the shortcut to AI citations that some marketing suggests, and treating it that way pulls attention from the factors that do make a difference: open crawler access, readable HTML, clear answers, and accurate information.
The most useful approach is to measure rather than assume. SEO Sets reviews AI readiness factors page by page and tracks how often major assistants cite a site over time, so teams can see which changes actually move the numbers. Add the file if it fits your site, then spend the bulk of the effort on access and content quality, where the evidence is far stronger.
