If you’re considering an llms.txt file to improve your Google rankings, it shouldn’t take priority over actual problems with your site. Google Search documentation says the file neither helps nor hurts search visibility, including in its generative AI features. Providing a useful index for an agent that consults your documentation is a different situation: there may be a specific reason to test it.
The question has become more confusing because Lighthouse also has an audit named llms.txt. That looks contradictory until you separate what each product evaluates. A check for agentic browsing does not make the file an SEO requirement.
What llms.txt is
It is a Markdown file that introduces a site or project and organizes links to relevant information. Think of it as an annotated documentation index: it suggests a reading path while the details remain on the linked pages.
The proposal maintained by Jeremy Howard was published in September 2024 and revised in August 2026. It remains a proposal, not a universal requirement for websites.
The intended reader matters. Publishing the file does not require an assistant to consult it, follow its links, or cite your company. Before installing a generator, ask: which system will use this index, and for what task?
Why Google Search and Lighthouse seem to disagree
Google’s guide to generative AI features limits its guidance to Google Search. In its section addressing misconceptions about AI optimization, it explains that Search ignores these files. Maintaining one for another system is fine, but it provides no ranking advantage in Google.
The Lighthouse llms.txt audit documentation, meanwhile, belongs to agentic browsing: agents interacting with websites. It describes two relevant behaviors: a server error when retrieving the file is flagged; a 404 response when the file does not exist is marked “Not applicable,” because providing it is optional.
An alert in this audit therefore needs to be read within its scope. It does not demonstrate a loss of search rankings or prove that publishing the file will make an AI recommend the business.
Comparing the documentation to make a decision
This analysis compares the two official documentation pages and the format proposal, consulted on October 31, 2026. Each source was assessed against the same question: what does it allow us to conclude about whether the file is needed?
| Source examined | Finding | Decision it supports |
|---|---|---|
| Google Search Central | Google Search ignores the file. | Do not justify implementing it as a way to improve Google rankings. |
| Lighthouse, agentic browsing | A missing file returning 404 is not applicable; a server error is flagged. | Investigate a file delivery error separately from deciding whether to provide the file. |
| llms.txt v2 proposal | An index can point agents toward relevant content. | Evaluate the format when a specific system and task have been identified. |
This comparison avoids two prioritization mistakes. The first is turning every tool warning into an SEO emergency. The second is dismissing a documentation resource just because it does not influence Google. The purpose of the work determines how to evaluate it.
A public example: the FastHTML documentation index
The public FastHTML file shows the proposal in use. When consulted, it introduced the project, noted usage precautions, and grouped links into documentation, API, examples, and optional material.
One detail is especially useful: before the links, the text warns that the library is not compatible with FastAPI syntax. That guidance aims to prevent a specific implementation error. Its proposed value is helping a person or agent use a tool correctly.
There is also a link to a concise Markdown guide. The file acts as an entry point to that material instead of combining all the documentation into a single block.
This inspection establishes how the public file is organized. It does not measure how many agents consult it, whether they make fewer errors, or whether FastHTML gets more traffic because of it. Those findings would require a different study. The author has no professional relationship with the project analyzed.
When to add the file to your work queue
If you manage a small business website and important pages are still hard to reach, information is outdated, or navigation is broken, start with those problems. An additional index would have to keep up with the same changes and could become another place where information falls out of sync.
For extensive technical documentation, the decision may be different. If your team already uses agents to consult reference material, it is worth testing a short index with a narrowly defined task, such as finding the correct authentication instructions. Decide who maintains the file and how links will be checked after changes.
Priority depends on the goal. Summary of official sources consulted on October 31, 2026; detailed criteria appear in the text.
To decide whether to run a pilot, I would ask three questions:
- Is there an identified system that will use it? Name the agent, integration, or workflow expected to read the index. “AI systems” is too broad to guide a test.
- Is there an observable problem? Record the current difficulty, such as consulting an outdated reference or failing to find a necessary instruction.
- Can it be maintained? Assign an owner and define how links and descriptions will be updated alongside the documentation.
If you do not have those answers yet, keep implementation as a hypothesis. You can revisit it when a concrete need arises.
llms.txt, robots.txt, and sitemaps serve different purposes
robots.txt provides crawling instructions. It does not protect private information and, on its own, does not guarantee that a URL will stay out of Google results.
A sitemap helps search engines discover URLs and provides information about files on the site. Having one does not guarantee that every item will be crawled or indexed.
An llms.txt file offers context and selected links to systems that choose to use it. It does not replace these mechanisms. Do not use it as access control, either: publishing an instruction for an agent does not protect the content it points to.
How to test without confusing access with outcomes
A useful pilot starts with a task and a verifiable answer. Choose a question whose solution is in the documentation, record the agent version, and compare equivalent runs with and without the index. If the workflow lets you provide the file manually, record that: handing the index to an agent does not demonstrate automatic discovery.
Keep the pages consulted, the final answer, and the errors found. Repeat the task, because individual runs can vary. A request for the file in server logs establishes that a request occurred, but does not prove useful reading, a citation, or a sale. Nor does an increase in traffic after publication isolate the effect of this change.
This article presents a documentation analysis and a proposed evaluation procedure. No experiment measuring ranking gains or agent performance was conducted. The guidance and proposal may change; revisit the sources before turning this conclusion into a permanent requirement.
This topic belongs to Technical and international SEO, which covers crawling, indexing, and site architecture.
To learn about my work in web development, technical SEO, and data, see my professional profile. If your team is deciding what to prioritize on its website, let’s talk on WhatsApp.
Documentation analysis produced with AI assistance. Sources consulted on October 31, 2026. The cover is an illustrative composite.