Cloudflare AI Crawler Controls: Build a Clear Access Policy

Cloudflare’s crawler controls help site owners distinguish search access, AI training preferences and agent traffic. The policy you choose should reflect how you want your public content used.

This guide reviews the controls described as of September 18, 2026, including Disallow AI Training, Bot Preference Sync and the Accountable designation for qualifying mixed-use crawlers. The practical task is to align your settings, published preferences and actual server responses. Check each layer so a training decision has the intended effect on search access.

Crawling is no longer one thing

The basic problem is that automated access to a website now serves very different purposes.

A crawler might be collecting pages for:

• traditional search results,
• AI-generated answers,
• an AI agent acting on behalf of a user,
• model training,
• retrieval and grounding,
• or some combination of the above.

Cloudflare now separates much of this activity into broad categories such as Search, Training, and Agent.

That sounds like a taxonomy change. It is more important than that. It means the old choice between “let the bot in” and “block the bot” is beginning to break down.

And that is probably a good thing.

Being found and being trained on are not the same decision

This is the most important change in how website owners should think about AI crawlers. Historically, some operators and crawler systems have served multiple purposes.

That creates an uncomfortable choice for publishers:

Allow the crawler and potentially allow more AI use than you want.

Or:

Block it and potentially sacrifice useful discoverability.

The ecosystem is increasingly separating those decisions. OpenAI gives publishers separate controls for OAI-SearchBot and GPTBot. Google provides Google-Extended as a separate control for certain Gemini-related uses while saying it does not affect Google Search inclusion or ranking.

Cloudflare now gives site owners separate policy controls for Search, Training, and Agent traffic. Its September 15 update also makes the training choice more granular: Disallow AI Training publishes the no-training preference while allowing Accountable mixed-use crawlers to remain available for search.

That gives businesses an increasingly important option:

You may want the referral without donating the textbook.

For many brands, that is a much more sensible conversation than either “block all AI” or “let everything eat.”

robots.txt is quietly becoming a policy document

Traditionally, robots.txt has mostly answered one question:

Where may a crawler go?

The AI era is pushing another question into the same neighborhood:

What may a crawler do with what it finds?

Cloudflare has been experimenting with and deploying machine-readable Content Signals that distinguish uses such as:

• search,
• AI input or grounding,
• and AI training.

Its newer content-use concepts go further by describing how retrieved information may be retained or reused. That is a meaningful change in philosophy.

The future robots.txt may not simply say:

Stay out of this folder.

It may effectively say:

• You can read this.
• You can index this.
• You can reference this.

But training is a separate question. There is an important caveat. These newer signals are still part of an evolving ecosystem. They are not a universal contract every AI company automatically obeys.

Cloudflare itself makes a related distinction very clearly:

robots.txt communicates preference. Infrastructure enforcement actually blocks requests.

So think of it like this:

robots.txt states the house rules. Your firewall determines whether somebody actually gets through the door. Those are related jobs. They are not the same job.

Cloudflare is trying to stop those two layers from disagreeing

Cloudflare’s newer bot preference tooling is interesting for exactly this reason. It can align choices made inside Cloudflare with the instructions exposed through robots.txt.

Without alignment, a website can accidentally say:

• robots.txt: Welcome.
• Firewall: Absolutely not.

Or:

• robots.txt: Please stay out.
• Infrastructure: Door is wide open.

Mixed signals. For technical SEO teams, this matters because crawler configuration is becoming less of a static text-file chore and more of an infrastructure decision. The person editing robots.txt may no longer be the only person controlling what machines can actually access.

So where does llms.txt fit?

This is where we should resist the urge to make every new file sound like the next sitemap.xml. llms.txt is almost the opposite of an access-control file.

It does not primarily say:

“Do not use this.”

It says something closer to:

“If you are trying to understand this website, here are the useful bits.”

The current llms.txt proposal gives a site a predictable Markdown resource containing context and curated links to important material. The 2026 v2 proposal also expands ideas around scoped files, clean Markdown alternatives, and discoverability.

That is interesting. Its value for a particular website depends on which systems use it and what task it supports. Google now explicitly says llms.txt does not help or hurt visibility or ranking in Google Search.

That is useful clarity. There can still be legitimate value for agents and systems that support the convention.

Particularly:

• documentation sites,
• technical knowledge bases,
• large product help centers,
• and information-dense resources.

But machine accessibility is not the same as AI visibility.

A beautiful Markdown directory cannot compensate for:

• weak content,
• ambiguous entities,
• bad internal linking,
• inconsistent business information,
• broken pages,
• or content that never answers anything useful.

llms.txt can organize the library. It does not write the books.

This is where SEO and GEO start sharing a desk

Traditional SEO asks whether search engines can:

• crawl,
• understand,
• index,
• and surface your website.

AI search adds questions such as:

• Which machine is accessing it?
• What purpose does the crawler serve?
• Can a retrieval system extract the critical information?
• Are entities clear?
• Is the correct source current?
• Can an agent operate the interface?

Those are extensions of technical SEO. Not a replacement for it.

Google’s current generative Search guidance says exactly that in its own way:

• Foundational SEO still matters.
• Its generative Search features rely on core Search systems.
• There is no special AI schema requirement.
• There is no Google benefit from llms.txt.
• There is no need to rewrite the entire website into strange machine chunks.

This is good news. The website does not need to become weird. It needs to become clear.

The interesting strategy is not “block AI”

For most businesses, we do not think the useful conversation is:

Should we block AI?

That is too crude. Imagine a law firm, medical practice, SaaS company, agency, or home-services business.

That company may very much want an AI assistant to know:

• where it operates,
• what services it provides,
• who works there,
• which problems it solves,
• what its hours are,
• and which pages explain a topic particularly well.

If somebody asks an assistant for a provider matching those criteria, being understood can be extremely valuable. But the same company may have a separate opinion about model training. Those are different decisions.

Cloudflare’s direction is significant because infrastructure is beginning to acknowledge that nuance.

The long-term strategy looks less like “AI optimization” and more like:

AI governance plus discoverability.

Be easy for the right systems to understand. Be explicit about how content should be used. Enforce decisions that actually matter. That is a much healthier framework than installing llms.txt and declaring victory.

Should most service businesses allow training?

Our default answer is usually yes. With a large asterisk. We do not have evidence that allowing training improves AI-search rankings or citation rates. A visibility benefit from training access remains unproven.

Our reasoning is strategic. If you are a newspaper, research publisher, licensed database, or company with valuable proprietary documentation, training access may raise meaningful rights and business-model questions. A roofer’s public guide to heat-damaged shingles serves a different commercial purpose from a paid research archive. Consider that distinction when setting permissions.

The content exists because you want the market to understand your expertise. In that case, blocking training may provide little practical benefit. The key is making the decision consciously. Do not confuse the ability to restrict training with a reason to restrict training.

The next wrinkle is agents

Training bots get most of the attention. Agents may ultimately be more disruptive. An AI agent does not necessarily visit a website to build an index or train a model.

It may visit because a human asked it to do something. Compare products. Read documentation. Check availability. Fill out a form. Book an appointment. Cloudflare now treats Agent as a distinct traffic category.

Google has published guidance discussing browser agents that inspect screenshots, DOM structure, and accessibility information. OpenAI similarly recommends accessible interface semantics for its browser-agent experiences. This tells us where the web is going.

The next version of the web may have:

• human visitors,
• search crawlers,
• training crawlers,
• retrieval systems,
• and software agents acting for people.

They will not all want the same thing. You probably should not treat them all the same way.

So what should a business actually do?

Evaluate each file against a documented use case and a system that supports it. Start by deciding what you actually want.

For many businesses, a reasonable direction is:

• Allow traditional search crawling.
• Allow AI search crawlers where discoverability matters.
• Make deliberate training decisions instead of inheriting defaults forever.
• Understand which AI and agent crawlers actually visit.
• Keep robots.txt aligned with technical enforcement.
• Maintain strong structured data, internal architecture, and entity signals.
• Keep canonical business facts current.
• Consider llms.txt where there is genuinely useful structured information to expose.
• Monitor how all of those pieces evolve.

Because they absolutely will. Cloudflare’s changes are not important because everybody suddenly needs three new toggles.

They are important because one of the largest infrastructure companies on the web is acknowledging something bigger:

Bot traffic is no longer a single category. Neither is discoverability. A website increasingly needs to decide not only whether machines can see it, but how different machines should interact with what they see.

SEO got us very good at inviting robots inside. The next chapter is figuring out what they are allowed to do once they get here.