AI Crawlability: How Search Engines, AI Crawlers and Retrieval Systems Actually Access Your Website

Crawlability starts with what your server returns to a request. To prepare a site for AI search, check both the visitor’s purpose and the content it can access.

Your website may receive requests from search crawlers, model-training crawlers, user-triggered fetchers, browser agents and monitoring tools. A user-agent name alone is insufficient to verify the operator.

Work through five questions:

• Who is making the request, and can you verify the identity?
• What access does your policy allow?
• Does the server or firewall return the intended page?
• Which facts require JavaScript or a user interaction?
• Can the crawler discover the links to other important pages?

These checks separate a policy decision from a delivery problem, so you can fix the right layer.

What AI crawlability actually means

We use “AI crawlability” as shorthand for a larger concept:

Can the systems involved in AI-assisted discovery reliably access the information they need from your website?

A successful HTTP response from one crawler is only one part of the check. Different AI products can obtain information in different ways. Some crawl directly. Some use search indexes.

Some rely on partner search providers. Some retrieve URLs at query time. Some launch browser-like agents. Some may access information through structured feeds or platform integrations rather than ordinary web crawling.

So the goal is not to memorize a list of bot names and call it a strategy. The goal is to make intentional decisions about machine access.

The five access layers

When we audit crawlability, we think about five layers.

1. Discovery

Does the system know the URL exists?

Discovery can happen through:

• internal links,
• sitemaps,
• external links,
• search indexes,
• feeds,
• APIs,
• submitted URLs,
• and previously discovered pages.

A page nobody links to may technically be crawlable while still being extraordinarily good at hiding. This is why internal linking belongs in technical architecture, not merely content strategy.

2. Permission

Is the crawler allowed to request the URL according to your published preferences?

This is where robots.txt appears. robots.txt can tell compliant crawlers which areas should or should not be crawled. It is a protocol for expressing preferences. It is not a force field.

Cloudflare makes this distinction explicit in its AI crawler documentation. A robots.txt directive expresses the site owner’s preference. If actual enforcement is required, Cloudflare recommends using technical controls such as AI Crawl Control.

That difference matters enough to repeat:

• robots.txt asks.
• Your infrastructure can enforce.

3. Network access

Even if robots.txt says yes, the request still has to survive the trip.

A crawler can be blocked by:

• a web application firewall,
• a CDN rule,
• bot protection,
• rate limiting,
• IP restrictions,
• geo restrictions,
• authentication,
• security plugins,
• server configuration,
• or the classic “we turned this on two years ago and nobody remembers why” rule.

Check for conflicting access controls:

• robots.txt: Welcome!
• Cloudflare WAF: Absolutely not.

From the website owner’s perspective, the crawler is “allowed.” The crawler receives a block instead of the page.

4. Response

Suppose the request reaches the server.

What comes back?

A healthy crawl response generally means the crawler receives the expected status code and the expected content.

But real websites return all sorts of surprises:

• 200 pages that are actually error messages,
• 403 responses to specific user agents,
• 302 chains,
• JavaScript application shells,
• cookie walls,
• interstitials,
• empty HTML waiting for an API call,
• or an entirely different version of the page based on geography.

A browser user may never notice. Machines absolutely can.

5. Rendering

Rendering introduces another dependency to check. The HTML response may contain the full article.

Or it may contain:

• a header,
• a footer,
• a root div,
• and a sincere promise that JavaScript will handle the rest.

Google can render JavaScript and documents a crawl, render, and indexing pipeline. Not every automated system behaves like Google. Google itself notes that server-side or pre-rendering is still useful because not all bots can run JavaScript.

This creates an important technical principle for AI search:

Critical information should not depend on advanced rendering behavior unless there is a good reason.

The crawler categories that actually matter

Instead of memorizing dozens of bots, begin by understanding the jobs they perform.

Search crawlers

These exist primarily to discover and index information used in search or search-like retrieval. Examples can include conventional search engine bots and AI-search-specific crawlers such as OpenAI’s OAI-SearchBot. OpenAI tells publishers that allowing OAI-SearchBot helps content be discovered, surfaced, and clearly cited in ChatGPT search.

If ChatGPT search visibility matters to you, blocking it without a reason deserves investigation.

Training crawlers

These collect web content that may be used to improve or train models. OpenAI’s GPTBot is the obvious example because OpenAI allows publishers to control it independently from OAI-SearchBot. This separation is extremely important.

You can have a conversation about model training without automatically having the same conversation about search discoverability.

Agents

Agents act on behalf of a user.

That can mean:

• opening pages,
• reading information,
• comparing products,
• checking availability,
• filling forms,
• or interacting with a website through a browser-like environment.

Cloudflare now treats Agent as a separate AI traffic category alongside Search and Training. Google’s own generative Search guidance discusses browser agents that may inspect screenshots, DOM structure, and accessibility information.

Agents are interesting because the old crawler model starts to blur. A crawler usually wants information. An agent may want to do something. We expect this distinction to become increasingly important.

User-triggered fetchers

Some systems fetch a page because a human explicitly requested it. That is different from crawling the site proactively for an index. It may also have different user-agent behavior, rate patterns, and robots semantics depending on the provider.

Again:

purpose matters.

A bot name is not an identity

User-agent strings are just text supplied by the requester.

Anyone can write:

Mozilla/5.0 DefinitelyGooglebotWePromise

This is why serious bot verification may involve:

• published IP ranges,
• reverse DNS verification,
• Cloudflare bot identification,
• or other provider-specific mechanisms.

Do not build a crawler policy based entirely on whether a request contains the letters “GPT.” A user-agent match alone does not verify the requester.

The JavaScript problem is not “JavaScript bad”

JavaScript is not bad.

The technical question is:

Which information are you making dependent on it?

A product carousel can depend on JavaScript. An animation can depend on JavaScript. A pricing calculator may reasonably depend on JavaScript. The primary service offered by the company probably should not exist only after JavaScript calls an API.

The official office address probably should not be injected exclusively through Tag Manager. The only crawlable link to a major practice area probably should not appear after a user opens a custom menu widget.

The distinction is not ideological. It is architectural.

What Google teaches us about rendering

Google is useful here because it publicly documents its JavaScript processing.

At a simplified level:

• Google crawls the page.
• It parses the initial HTML.
• Eligible pages can enter a rendering queue.
• Google’s rendering system executes JavaScript.
• Google processes the rendered HTML.

That means Google may ultimately see content that did not exist in the original response. It also means the original response and rendered page are distinct technical states. That distinction is useful beyond Google.

When we analyze AI accessibility, we want to know which facts exist at each state. Initial response. Rendered DOM. Interaction-dependent state. Third-party widget state. Those layers tell us how fragile machine access might be.

robots.txt is getting a bigger job

robots.txt was historically mostly about crawl access. The AI era is pushing more nuance into the same neighborhood.

Different providers expose ways to distinguish:

• search,
• training,
• AI use,
• and agents.

Cloudflare is also building controls around AI crawler behavior and robots compliance. This does not mean robots.txt has transformed into a universal AI licensing contract. It has not. But website owners now need to treat crawler policy as an intentional configuration.

The days of copying the same robots.txt file between every website and never thinking about it again are ending.

Search access and training access should be separate decisions

OpenAI makes the distinction easy to understand. OAI-SearchBot relates to ChatGPT search discovery and citation. GPTBot can be controlled separately for potential training use.

That means a business can decide:

• Yes, I want ChatGPT search to discover this.
• No, I do not want GPTBot crawling it for training.

Or:

Yes to both.

Or:

No to both. The correct choice depends on the business. What we object to is making the choice accidentally.

Cloudflare makes the infrastructure layer visible

Cloudflare’s AI Crawl Control is important because it shows what crawler policy looks like when it leaves the text file and enters actual network enforcement.

Cloudflare says AI Crawl Control can:

• show which AI services access the site,
• display request patterns,
• apply crawler-specific allow or block policies,
• monitor robots.txt compliance,
• and enforce behavior at the edge.

This creates a useful two-layer model:

• Policy says what you want.
• Infrastructure determines what happens.

A mature AI search audit should understand both.

What about llms.txt?

llms.txt belongs nearby, but it is not an access-control system. It is better thought of as a proposed machine-friendly map or curated resource.

robots.txt says something like:

Here is what you may crawl.

llms.txt says something closer to:

If your system supports this convention, here are the resources we think are useful.

Those are different jobs. Google currently says llms.txt does not help or hurt visibility in Google Search. That alone should stop people from treating it like robots.txt 2.0. We cover it separately because there are legitimate use cases, particularly for documentation-heavy sites.

How internal links affect crawlability

Crawler access is not only about blocking. Machines need paths. A crawlable link should generally be a normal HTML link with a resolvable destination.

If a page is reachable only through:

• a JavaScript event,
• a button without a useful link,
• an internal search form,
• or some custom interface state,

discovery can become more fragile.

Good internal linking does three jobs at once:

• It helps users navigate.
• It helps crawlers discover pages.
• It communicates relationships between pages.

That is why our AI-search content architecture treats internal linking as part of machine understanding, not merely “SEO juice.”

Crawlability is not indexability

Another distinction worth preserving. A page can be crawlable and still not be indexed. A page can be indexed and not be selected for a generative answer. A page can be selected as a source and not receive a visible citation.

A page can receive a citation and no click. These are separate stages. Treating them as one metric is how people end up blaming robots.txt for content problems.

Status codes still matter

AI did not repeal HTTP. 200 still means something. 301 still means something. 404 still means something.

A crawler repeatedly finding:

• redirect loops,
• soft 404s,
• server errors,
• temporary failures,
• or inconsistent canonical responses

is not participating in a futuristic AI mystery. The site is technically messy. Address these access and response problems first.

What should service businesses allow?

There is no universal configuration.

But for an ordinary service business that wants discoverability, our starting posture is generally:

Allow major search crawlers. Allow AI-search crawlers where visibility is desired. Evaluate training separately. Avoid blanket bot blocking unless there is a clear reason. Keep critical public information accessible. Use enforcement where actual blocking matters.

Monitor what is hitting the site. Then adjust. A law firm, plumbing company, agency, accountant, or dental office usually publishes service information because it wants people and systems to understand what the business does.

That does not mean every automated use should be allowed forever. It does mean reflexively blocking everything with “AI” in the name can be counterproductive.

The crawlability checklist

For important public pages, review: