robots.txt for AI: Search Crawlers, Training Bots and Agents Are Not the Same Thing

A crawler policy should reflect what you want each system to do with your content. Search indexing, retrieval for AI answers, model training and user-directed browsing serve different purposes.

Start by identifying the relevant crawlers and their documented controls. Then compare your robots.txt preferences with the rules enforced by your hosting, firewall and bot-management tools. This guide explains those layers and shows how to check that your policy matches the response a crawler actually receives.

The most important distinction

If you remember only one thing from this article, make it this:

Search access does not automatically have to equal training permission.

OpenAI makes this unusually easy to see. OAI-SearchBot is associated with discovery and citation in ChatGPT search. GPTBot can be controlled separately for content site owners want excluded from potential training.

Google makes a similar separation through Google-Extended. Google says the Google-Extended token can control certain uses related to Gemini training and grounding without affecting inclusion or ranking in Google Search.

It means website owners can finally make a more sophisticated decision than:

“AI yes” or “AI no.”

The four questions a crawler policy should answer

For every major crawler or crawler category, ask:

1. Do we want this system to discover our public content?
2. Do we want this system to use our content in search or real-time AI answers?
3. Do we want this system to use our content for model training?
4. Do we want automated agents acting for users to interact with the site?

Those answers can differ. A publisher might want search visibility but reject model training. A documentation site might actively want agents to retrieve technical information. A private research platform might restrict almost everything.

A local law firm may have little reason to block public service information from any major legitimate crawler. The mistake is not choosing one answer over another. The mistake is never asking.

robots.txt is a preference layer

robots.txt is a public declaration of crawler preferences. That makes it useful. It does not make it enforcement. Cloudflare’s documentation says this plainly. robots.txt compliance is voluntary. If actual technical blocking is required, Cloudflare recommends enforcement controls such as AI Crawl Control.

Think of robots.txt as the sign. Think of your CDN, WAF, server, or crawler-control platform as the lock. A crawler that respects the sign behaves accordingly. A crawler that ignores it may still reach the door unless infrastructure stops it.

This distinction matters when legal, licensing, security, or cost concerns make access genuinely important.

How OpenAI splits search and training

OpenAI currently gives publishers a useful separation.

Those are different controls.

So a robots.txt policy could conceptually say:

• Allow the search crawler.
• Disallow the training crawler.

That does not guarantee a certain citation outcome. It simply avoids accidentally blocking a crawler associated with the search product you want visibility in. OpenAI also notes that ChatGPT referral links include utm_source=chatgpt.com, which gives publishers a practical way to measure some search traffic once visibility exists.

Google-Extended is not Googlebot 2

Google-Extended is especially misunderstood. It is a robots.txt product token. It is not a separate crawler with its own HTTP user-agent. Google says the token can be used to control whether content already crawled by Google may be used for certain Gemini-related model training and grounding use cases.

Google also says Google-Extended does not affect inclusion in Google Search and is not used as a Search ranking signal. That distinction is important. If your goal is to remain visible in Google Search while making a separate decision about some Gemini uses, Google provides a mechanism for doing that.

Again:

Search policy and AI-use policy do not have to be identical.

Google Search AI features still follow Googlebot controls

This is where things get nuanced. Google’s AI features inside Search are part of Google Search. Google says robots.txt directives for Googlebot remain the site-owner control for crawling in Search, including generative Search features.

Preview controls such as nosnippet, data-nosnippet, max-snippet, and noindex can also affect what appears from pages in Search. So do not assume every Google AI surface maps to Google-Extended. It does not.

Google Search’s AI features and other Gemini-related uses have different control paths. This is why “block AI” is not a technical specification.

Cloudflare is making purpose explicit

Cloudflare’s current AI crawler controls are useful because they organize traffic around what the crawler is doing.

Its broad categories include:

• Search
• Training
• Agent

Cloudflare also exposes AI Crawl Control for observing and enforcing crawler behavior. That is a more modern framing than maintaining a growing list of individual user agents forever. It does not remove the need to understand provider-specific controls.

It gives infrastructure teams another layer for expressing policy.

Content signals go even further

Cloudflare has also been extending managed robots.txt with Content Signals.

Its documentation describes signals such as:

• search
• ai-input
• ai-train
• and newer content-use concepts.

The idea is to communicate not only whether content can be accessed, but the intended category of use. This is an important direction. It is not yet a universal law of the web.

Neither is llms.txt, which is a different kind of file altogether. Treat these signals as part of an evolving ecosystem, not as a magical licensing protocol every crawler already obeys.

That distinction should appear in any serious implementation discussion.

So should you use wildcard AI block lists?

Be careful. A wildcard or blanket block can feel satisfyingly decisive. It can also block things you actually want. For a publisher protecting licensed content, broad restrictions may be intentional.

For a service business actively trying to be discovered in AI-assisted search, blocking every crawler labeled “AI” can work against the business goal. Start from the desired outcome. Then configure access.

Do not start from a giant copied disallow list and reverse-engineer the consequences later.

What about bad bots?

Legitimate crawler policy is only one part of bot management. Fake bots may spoof recognizable user agents. Scrapers may ignore robots.txt. Automated traffic can create performance or abuse problems independent of AI search.

That is why infrastructure enforcement belongs in the conversation.

Useful controls can include:

• verified bot detection,
• provider IP verification,
• rate limiting,
• WAF rules,
• behavioral bot management,
• and crawler-specific policies.

This is security and infrastructure territory, not merely SEO. SEO should still understand the consequences. A security rule that blocks OAI-SearchBot may be perfectly secure and perfectly counterproductive for a business that wants ChatGPT search visibility.

Teams need to talk to each other. Document the choice so it can be reviewed later.

robots.txt cannot fix content you should not have published

Another important distinction. robots.txt is not a confidentiality system. If content must be private, do not rely on crawler directives to protect it.

Use:

• authentication,
• authorization,
• proper access controls,
• and appropriate application security.

Publishing sensitive material publicly and adding Disallow is not security. It is a polite note attached to an unlocked filing cabinet.

What about noindex?

robots.txt and noindex do different things. A robots.txt disallow can prevent a compliant crawler from fetching the page. A noindex directive tells a crawler that can access the page not to index it.

This distinction matters because a crawler generally has to fetch a page to read its meta robots directive. OpenAI notes a similar nuance in its publisher guidance. If you want a crawler to observe page-level metadata, it must be able to access the page.

Do not casually combine broad disallows with page-level controls without understanding which instruction can actually be seen.

A good policy is simple enough to explain

For many service businesses, a starting policy may look like:

• Traditional search crawlers: allow.
• Major AI search crawlers: allow where discovery is desired.
• Training crawlers: make an explicit business decision.
• Agents: allow where useful and safe, with normal security controls.
• Known abusive automation: block or rate-limit.
• Private content: protect with real access controls.

That is not a universal configuration. It is a sane conversation starter.

How this should appear in a technical audit

A weak audit says:

“robots.txt exists.”

Great.

A useful audit says:

“OAI-SearchBot is disallowed in robots.txt while the business has stated ChatGPT search visibility is a priority. GPTBot is also disallowed. Because OpenAI separates search crawling from potential training, consider allowing OAI-SearchBot while keeping the GPTBot restriction if the training preference remains intentional.”

It connects:

• configuration,
• documented crawler purpose,
• business intent,
• and remediation.

That is the standard we want Lumen to move toward.