llms.txt: What It Does and When to Use It

An llms.txt file provides a curated guide to website content for systems that choose to use it. Its value depends on the consumer, the quality of the references and whether someone maintains it.

For a business website, start with a specific use case: helping a documentation assistant find the right guide, for example. Then check whether the intended system supports the file and whether it improves the task.

This article separates the file’s proposed role from search eligibility, compares it with other site files and provides a practical evaluation method.

What llms.txt actually is

llms.txt is a proposed convention for publishing concise, Markdown-formatted information intended to help language models and agents use a website.

The current proposal describes:

• a file at /llms.txt or a relevant subpath,
• a project/site name,
• a short summary,
• optional explanatory material,
• and lists of useful resources.

The proposal also encourages clean Markdown versions of important pages. The idea is sensible. Web pages contain enormous amounts of presentation machinery. Navigation. Scripts. Ads. Tracking. Interactive components. Markdown can offer a cleaner representation of the useful information.

For documentation and agent use cases, that can be genuinely helpful.

What llms.txt is not

It is not a universal AI-search ranking directive. It is not a replacement for robots.txt. It is not a sitemap replacement. It is not a schema replacement. It is not a guarantee that a model reads anything.

And Google currently says llms.txt does not help or hurt visibility or ranking in Google Search, including its generative AI features. That last point deserves a sticky note. If somebody sells you llms.txt specifically as a Google AI Overview ranking tactic, ask for the evidence.

Evaluate whether that evidence supports the proposed work.

Google’s position is refreshingly clear

Google’s 2026 generative Search guidance explicitly includes llms.txt in its myth-busting section. Google says it does not use llms.txt for Search. Creating the file will neither help nor hurt Search visibility.

That does not mean the format has no value. Google is one system. The llms.txt proposal targets a wider ecosystem of language models and agents.

The important lesson is narrower:

Do not confuse adoption by some tools with a universal ranking mechanism.

Where the format makes the most sense

Documentation. This is the easiest use case.

Imagine a developer agent trying to understand:

• an API,
• a software framework,
• an SDK,
• configuration options,
• or a large technical knowledge base.

A curated Markdown map can reduce navigation noise and point the agent toward canonical resources. The llms.txt proposal itself has evolved around this use case. Its 2026 v2 update adds better discoverability conventions for Markdown alternatives and scoped llms.txt files.

That is real utility. Now compare that with a five-page local service website. Home. About. Services. Contact. Blog. The need is less obvious. An agent does not necessarily require a special machine-readable index to understand that structure if the site is already clear.

This is why we prefer:

Use llms.txt where it solves a real retrieval problem.

Do not add it because your SEO plugin discovered another checkbox.

The v2 proposal is more interesting than a root file alone

The current proposal is not merely:

put links in /llms.txt.

It also discusses:

• scoped llms.txt files,
• Markdown alternatives to HTML pages,
• and discoverability relationships that can point an agent toward those machine-friendly versions.

That begins to look less like an SEO trick and more like content infrastructure for software agents. That is the right mental model.

The strongest use case may eventually be:

“Here is a clean, canonical, low-noise representation of our documentation.”

Not:

“Here is the file that ranks us in ChatGPT.”

llms.txt versus robots.txt

These files answer fundamentally different questions.

One is primarily a crawler-preference mechanism. The other is a proposed discovery and context mechanism. A website can reasonably have both. They do not replace each other.

llms.txt versus sitemap.xml

A sitemap is mainly about URL discovery and metadata for search crawlers. It can expose a large inventory of canonical URLs. llms.txt is curated. It is supposed to help a model or agent understand which resources are useful and what they represent.

A sitemap might list 40,000 documentation pages.

An llms.txt file might say:

• Start with authentication.
• Then API endpoints.
• Then error handling.
• Optional examples are over here.

Different job.

llms.txt versus structured data

Structured data expresses facts in a formal vocabulary.

For example:

• This entity is an Organization.
• This person works here.
• This is a LocalBusiness.
• This product has this offer.

llms.txt is closer to a curated orientation document. Schema describes entities and properties. llms.txt describes useful resources and context.

Again:

different job.

Should Mithril clients get one?

Sometimes.

We would prioritize it for:

• large documentation sets,
• technical SaaS sites,
• complex knowledge bases,
• sites actively serving software agents,
• and businesses with substantial structured educational resources.

For a normal service business, we would first prioritize:

• crawlability,
• strong service pages,
• clear location information,
• entity consistency,
• structured data where appropriate,
• internal linking,
• original useful content,
• canonical URLs,
• and measurement.

If those are strong, an llms.txt file may be a reasonable additional layer. It is not the foundation.

What should go in one?

If you create an llms.txt file, keep it useful.

Include:

• site/project name,
• clear short description,
• important canonical resource groups,
• descriptive link labels,
• brief notes where they reduce ambiguity,
• and only resources you genuinely want agents to use.

Avoid:

• keyword stuffing,
• hundreds of undifferentiated links,
• marketing fluff,
• invented facts,
• obsolete pages,
• URLs that redirect,
• and duplicating every sitemap URL.

This is an orientation document. Act like a helpful librarian. Not a doorway salesperson.

The clean Markdown idea may matter more than the file

One of the most interesting parts of the llms.txt ecosystem is the idea of exposing clean Markdown versions of content.

Why?

Because machines often do not need:

• navigation chrome,
• CSS,
• carousels,
• animation wrappers,
• ads,
• cookie banners,
• or fifteen nested divs to understand a paragraph.

A clean Markdown representation can reduce noise and make technical content easier to retrieve. That idea is particularly compelling for documentation and agents.

It also reinforces the bigger Mithril thesis:

Machine accessibility is not only about whether the URL returns 200.

It is about whether the useful information is easy to extract.

How this connects to internal linking

Do not let llms.txt become a substitute for good site structure.

If the file says:

“Here are our five core concepts”

but the website itself does not connect those concepts through navigation and contextual links, you have created a machine-readable map to compensate for a messy human-readable city. Fix both.

A strong information architecture benefits:

• people,
• traditional crawlers,
• retrieval systems,
• and agents.

llms.txt can be a supplemental guide. The site should still make sense without it.

The decision framework

Create llms.txt when:

• your site is complex enough to benefit,
• you have canonical resources worth curating,
• compatible agents matter to the business,
• and you can maintain it.

Do not prioritize llms.txt when:

• the underlying content architecture is broken,
• critical facts are inconsistent,
• the site has crawl/indexing problems,
• or you are doing it solely because somebody called it the “new sitemap.”