How to See Which AI Bots Actually Crawl Your Website

Request logs show which pages were requested, when the requests arrived and how your server responded. They are a practical starting point for checking crawler access and investigating the effect of a policy change.

Before changing robots.txt or crawler rules, collect a baseline. Record the claimed user agent, verified identity where available, requested path, status and request volume. Use those records to distinguish successful access from blocks, errors or unexpected traffic. Pair them with separate referral and conversion data when you assess business outcomes.

Crawler activity is not referral traffic

Start with the distinction that causes the most confusion. Crawler traffic is machine requests to your website. Referral traffic is a human or user-driven visit arriving from another service. Those are different datasets.

OAI-SearchBot crawling a page does not mean a person clicked from ChatGPT. A person arriving from a ChatGPT citation does not prove which crawler originally discovered the page. OpenAI currently adds utm_source=chatgpt.com to referral links from ChatGPT search, which makes that second category easier to measure.

Crawler logs and analytics should be viewed together, not combined into one imaginary metric.

Where to look

Depending on the site architecture, useful sources can include:

• server access logs,
• CDN logs,
• Cloudflare AI Crawl Control,
• Cloudflare bot analytics,
• managed hosting logs,
• application logs,
• security tools,
• reverse proxy logs,
• and analytics for actual referral traffic.

Search Console and Bing Webmaster Tools answer additional questions about search and citation visibility, but they are not raw crawler logs. Each source tells you a different part of the story.

What a log entry can tell you

At a basic level, an HTTP request can expose:

• timestamp,
• requested URL,
• HTTP method,
• status code,
• user agent,
• IP address,
• referrer in some cases,
• response size,
• response time,
• and other infrastructure-specific fields.

That lets you ask useful questions.

Did OAI-SearchBot request the page?

Did it receive 200 or 403?

Did GPTBot repeatedly hit a section you intended to block?

Did a crawler discover thousands of junk parameter URLs?

Is a known bot stuck in redirect chains?

Is your WAF challenging something robots.txt explicitly allows?

User-agent strings are not proof

This deserves a permanent place in crawler documentation. A user-agent string is supplied by the requester. It can be spoofed. A request claiming to be Googlebot is not automatically Googlebot.

A request claiming to be GPTBot is not automatically OpenAI. Verification methods vary by provider and infrastructure.

Depending on the crawler, you may use:

• published IP ranges,
• reverse DNS checks,
• forward-confirmed reverse DNS,
• Cloudflare verified-bot classifications,
• or provider-specific verification instructions.

The right method depends on the bot. Do not create allowlists based solely on a string somebody can type into curl.

What Cloudflare AI Crawl Control adds

Cloudflare AI Crawl Control is useful because it moves some of this work out of raw logs.

Cloudflare says the product can show:

• which AI services access the site,
• request patterns,
• crawler-specific traffic,
• robots.txt compliance,
• and policy enforcement.

For sites already on Cloudflare, this can dramatically reduce the effort required to understand AI crawler activity.

It also introduces an important distinction between:

• observing,
• requesting compliance,
• and enforcing.

Seeing a bot in the dashboard does not mean the bot is harmful. Blocking a bot does not mean the bot was causing a problem. Measure first. Then decide.

The paths matter more than the total

Suppose a crawler makes 10,000 requests.

Is that a lot?

Maybe.

What did it request?

If 9,000 requests are hitting:

• calendar URLs,
• faceted navigation,
• tracking parameters,
• internal search pages,
• or duplicate archives,

the problem may be crawl architecture.

If requests focus on:

• service pages,
• documentation,
• articles,
• location pages,
• and product resources,

the same volume tells a different story. Do not stop at bot totals. Look at the URL patterns. AI crawlers can become a useful mirror for technical clutter. If machines keep discovering garbage URLs, the garbage exists somewhere.

Status codes tell the next part

Pay attention to:

• 200
• 301/302
• 403
• 404
• 429
• 5xx

A known crawler receiving large numbers of 403 responses may be blocked by security infrastructure. 429 responses may indicate rate limiting. 5xx errors can indicate server instability. Chains of 301s may reveal old migration debt.

404 patterns can expose stale internal links or old content discovery. Again, none of this is uniquely “AI SEO.” Good technical foundations remain useful no matter which crawler appears next.

robots.txt compliance is observable

Cloudflare specifically highlights robots.txt compliance monitoring. That is valuable because robots.txt is voluntary. If a crawler repeatedly accesses disallowed paths, that is different from a crawler that stops. Do not assume.

Observe. For high-value publishers, compliance behavior may influence whether infrastructure enforcement is necessary. For ordinary service businesses, the exercise can also uncover accidental blocks of useful crawlers.

The request for robots.txt is itself useful

Known crawlers commonly request /robots.txt. Seeing that request can help establish that a crawler attempted to understand site policy. It does not prove every subsequent request complies. It does not prove the identity is legitimate.

But it is a useful event in context.

When debugging crawler access, look at the sequence:

• robots.txt request,
• target-page request,
• status returned,
• follow-up requests.

Patterns tell stories.

Crawlers can cost real money

This becomes more important for large sites.

Automated traffic consumes:

• bandwidth,
• server resources,
• rendering resources,
• API calls,
• database queries,
• and sometimes third-party service quotas.

A badly behaved crawler hitting expensive routes can create real infrastructure cost. That makes crawler policy more than an SEO discussion.

If an AI crawler is aggressively requesting low-value or expensive dynamic URLs, consider:

• crawl controls,
• rate limiting,
• URL cleanup,
• caching,
• or infrastructure rules.

But diagnose before blocking the entire operator. The expensive route might be the real bug.

How to distinguish crawling from human AI traffic

Use analytics and referrals for humans.

OpenAI says ChatGPT search referral links include:

utm_source=chatgpt.com

That can be measured in GA4 or another analytics platform. Other AI products may appear through referral domains, tagged links, direct-like traffic, or other mechanisms. Attribution is imperfect.

Still, keep two separate reports:

• AI crawler activity
• AI referral and conversion activity

One tells you machines visited. The other tells you humans arrived. That distinction is worth preserving forever.

What if you do not have Cloudflare?

You can still inspect server or hosting logs. Managed hosts often expose access logs. Reverse proxies may log requests. Security platforms may expose bot data. For smaller sites, even limited logs can answer basic questions about whether known crawlers reach important pages.

The exact implementation depends on the stack. The principle does not. Observe before you theorize.

The minimum AI crawler dashboard

If we were building a simple operational view, we would want:

• Verified crawler/provider
• Category: Search, Training, Agent, Other
• Requests over time
• Top URLs
• Status codes
• Robots.txt compliance
• Blocked/challenged requests
• Average response cost where available
• Last seen
• Policy status
• Last policy review

That is enough to make real decisions.