# Crawler vs fetcher vs AI browser vs search bot: the agent taxonomy

_2026-07-02 · agents, analytics_

"AI bot" is not one thing. The automated traffic hitting your site splits into at least four kinds: crawlers that read in bulk, fetchers that pull one page to answer a live question, agentic browsers that act on a person's behalf, and search bots that index for answer engines. Each behaves differently and means something different for you. In 2025, automated traffic grew about eight times faster than human traffic ([HUMAN Security](https://www.globenewswire.com/news-release/2026/04/09/3270682/0/en/human-security-s-2026-state-of-ai-traffic-cyberthreat-benchmark-report-signals-a-new-internet-era-automation-growth-now-outpaces-humans.html), 2026), so the difference is worth getting right.

> **Key takeaways**
>
> - There are four distinct types of AI bot: crawlers, live fetchers, agentic browsers, and search or answer-engine bots. Lumping them together hides the story.
> - A crawler reading you thousands of times means something very different from one fetcher pulling a page to answer a buyer's question.
> - Crawl-to-referral ratios vary wildly: ClaudeBot has been measured near 24,000 crawls per visit it sends back, against single digits for classic search (Cloudflare, 2025).
> - Agentic browsers are the hard case. They drive a real browser and present an ordinary Chrome or Safari user-agent, so user-agent rules miss them by design.

## What are the four types of AI bots?

The four types are crawlers, fetchers, agentic browsers, and search bots, and they differ on two axes: whether they read in bulk or in real time, and whether they identify themselves or hide inside a normal browser. Crawlers and search bots gather at scale and name themselves. Fetchers act in real time and name themselves. Agentic browsers act in real time and do not.

Here is the practical split before we go deeper on each:

- **Crawlers** read content in bulk, usually for training or indexing. They name themselves with a token like `GPTBot`.
- **Fetchers** pull a single page live to answer a question someone just asked. Examples include `ChatGPT-User` and `Perplexity-User`.
- **Agentic browsers** drive a real browser session on a person's behalf, rendering JavaScript like a user would.
- **Search bots** index your pages so an answer engine can cite or rank them, like `OAI-SearchBot`.

Caprail classifies every request into one of these on the server, because the category changes what the visit means. For the wider context, see [analytics for AI agents](/blog/analytics-for-ai-agents).

## What is an AI crawler, and which user-agents are they?

A crawler is a bot that reads your pages in bulk, typically to build a training set or an index. It does not wait for a human prompt. It works through your site systematically, and the large ones run no browser, so they execute no JavaScript. That is why a browser analytics tag never records them, a gap we cover in [server-side vs client-side analytics](/blog/server-side-vs-client-side-analytics).

These are the crawler tokens worth knowing in 2026:

| User-agent token     | Operator     | Purpose                          |
| -------------------- | ------------ | -------------------------------- |
| `GPTBot`             | OpenAI       | Training data collection         |
| `ClaudeBot`          | Anthropic    | Training data collection         |
| `CCBot`              | Common Crawl | Open dataset used by many models |
| `Bytespider`         | ByteDance    | Training crawler                 |
| `Amazonbot`          | Amazon       | Index for Amazon AI services     |
| `Meta-ExternalAgent` | Meta         | Training crawler                 |
| `cohere-ai`          | Cohere       | Training crawler                 |

Two tokens you will see in robots.txt advice are not crawlers at all. `Google-Extended` and `Applebot-Extended` send no requests. They are opt-out signals that tell Google and Apple whether content they already fetched may train a model. Blocking them changes training permission, not crawl traffic.

## What is an AI fetcher, and how is it different from a crawler?

A fetcher reads one page live, at the moment a person asks a question the assistant cannot answer from memory. The difference from a crawler is timing and intent. A crawler reads broadly and ahead of time. A fetcher reads narrowly and on demand, because a specific human is waiting for the answer right now.

These are the main fetcher tokens:

| User-agent token       | Operator   | Triggered by                |
| ---------------------- | ---------- | --------------------------- |
| `ChatGPT-User`         | OpenAI     | A ChatGPT user's prompt     |
| `Claude-User`          | Anthropic  | A Claude user's question    |
| `Perplexity-User`      | Perplexity | A Perplexity user's query   |
| `meta-externalfetcher` | Meta       | A user-initiated link fetch |

The intent gap is the whole point. A fetcher visit means a real person is mid-conversation and your page was relevant enough to pull. That is a high-intent signal, closer to a search click than to a bulk crawl. Fetcher traffic has also grown fastest: Cloudflare measured `ChatGPT-User` requests up roughly 2,825 percent over a single year ([Cloudflare](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/), 2025).

## What is an agentic browser, and why is it hard to detect?

An agentic browser drives a real browser session on a person's behalf, clicking, reading, and rendering JavaScript the way a user would. Tools like OpenAI's ChatGPT Atlas, Perplexity's Comet, and Claude for Chrome fall in this group. They are the hardest category to measure, and the reason is structural.

Because they run an actual browser, they present that browser's ordinary user-agent, a normal Chrome or Safari string, with no distinct AI token at the request level. So a detection rule that looks for `GPTBot` or `ChatGPT-User` finds nothing. The agentic browser looks exactly like a person, by design. This is the one type that does run your JavaScript, which means it can land in browser analytics, just mislabeled as human.

What can you do about it? Honestly, less than with the named bots, and that is worth stating plainly rather than pretending otherwise. Behavioral signals and request patterns help at the margin, but no user-agent rule will reliably separate an agentic browser from a human. As this category grows, "is this visitor a person" becomes a genuinely fuzzy question.

## What is a search or answer-engine bot?

A search bot indexes your pages so an answer engine can cite or rank them in its results. It overlaps with crawlers, but the goal is different. A training crawler reads you to learn from you. A search bot reads you to find you later, when a user's query should surface your page. The distinction matters because indexing is how you get cited.

The key indexing tokens:

| User-agent token   | Operator   | Role                             |
| ------------------ | ---------- | -------------------------------- |
| `OAI-SearchBot`    | OpenAI     | Indexes pages for ChatGPT search |
| `Claude-SearchBot` | Anthropic  | Indexes pages for Claude search  |
| `Googlebot`        | Google     | Classic search index             |
| `Applebot`         | Apple      | Index for Siri and Spotlight     |
| `PerplexityBot`    | Perplexity | Index for Perplexity answers     |

OpenAI alone runs three separate bots that people routinely confuse: `GPTBot` trains, `ChatGPT-User` fetches live, and `OAI-SearchBot` indexes. Block the wrong one and you can quietly remove yourself from ChatGPT's search results while leaving training untouched. Knowing which token does what is the difference between governing your content and breaking your own visibility. We mapped one quarter of this traffic in [the Q2 2026 crawler share report](/blog/ai-crawler-share-q2-2026).

## How much does each type of AI bot send back?

Very different amounts, and the ratio is the most revealing number you can track. The crawl-to-referral ratio measures how many times a bot reads you for each human visit it sends back. Bulk crawlers read enormously and refer rarely. Classic search refers far more per crawl.

| Crawler       | Crawls per referral sent |
| ------------- | ------------------------ |
| ClaudeBot     | ~24,000 to 1             |
| GPTBot        | ~1,276 to 1              |
| PerplexityBot | under 200 to 1           |
| Google search | ~3 to 30 to 1            |
| DuckDuckGo    | ~1.5 to 1                |

Source: Cloudflare Radar 2025 Year in Review. These are network-wide figures; your own ratio will differ.

A high ratio is not automatically bad. It can mean an assistant is learning your product in depth before it ever sends someone your way. But it is a real cost, in bandwidth and in content given away, and you can only weigh it when you count both sides in one place. Network averages tell you the shape. First-party data tells you yours.

## Which AI bots generate the most traffic right now?

A handful dominate, though the order shifts month to month. By crawler share, Cloudflare measured Googlebot around 27 percent, GPTBot near 11.5 percent, Bytespider about 10.3 percent, and ClaudeBot around 9.7 percent in May 2026 ([Cloudflare](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/), 2026).

| Crawler    | Share of crawler traffic |
| ---------- | ------------------------ |
| Googlebot  | ~27%                     |
| GPTBot     | ~11.5%                   |
| Bytespider | ~10.3%                   |
| ClaudeBot  | ~9.7%                    |

Source: Cloudflare, May 2026. These shares flip month to month, so date-stamp them whenever you cite them.

Grouped by operator, HUMAN Security attributed roughly 69 percent of AI-driven traffic to OpenAI's bots, about 16 percent to Meta, and around 11 percent to Anthropic in its 2026 benchmark ([HUMAN Security](https://www.globenewswire.com/news-release/2026/04/09/3270682/0/en/human-security-s-2026-state-of-ai-traffic-cyberthreat-benchmark-report-signals-a-new-internet-era-automation-growth-now-outpaces-humans.html), 2026). Different networks measure different mixes, so treat any single share as a snapshot, not a constant.

## How do you tell these bots apart on your own site?

You read the request on the server and classify the user-agent, because that is the only layer every type passes through. A browser tag misses the headless crawlers and fetchers entirely, since they run no JavaScript. The server sees all of them, plus the format you served each one. Caprail does this classification automatically, sorting each request into crawler, fetcher, browser, search bot, or human.

The one type this cannot fully catch is the agentic browser, since it wears a normal browser's user-agent. That is a limit of the medium, not of any single tool, and it is better to know the boundary than to trust a number that quietly excludes it. For everything else, server-side classification turns a vague "we get AI traffic" into a labeled feed you can actually act on. See [how the markdown twin works](/blog/hello-agentic-web) for the next step once you can see the agents.

## FAQ

### What is the difference between an AI crawler and an AI fetcher?

A crawler reads pages in bulk, ahead of time, usually for training or indexing, like `GPTBot`. A fetcher reads one page live when a person asks a question, like `ChatGPT-User`. The crawler is systematic and impersonal. The fetcher is on-demand and high-intent, because a real human is waiting for the answer.

### Are Google-Extended and Applebot-Extended crawlers?

No. They send no requests and crawl nothing. Both are opt-out tokens in robots.txt that tell Google and Apple whether content they already fetched may be used to train their models. Blocking them changes training permission only. The actual crawling is done by `Googlebot` and `Applebot`, which are separate.

### Why can't I detect agentic browsers like ChatGPT Atlas or Comet?

Because they drive a real browser and present an ordinary Chrome or Safari user-agent, with no distinct AI token. A detection rule looking for known bot tokens finds nothing, since the request looks human. They also run JavaScript, so they can show up in browser analytics, just mislabeled as a person rather than an agent.

### Should I block AI bots in robots.txt?

It depends on the type, which is exactly why the taxonomy matters. Blocking a training crawler like `GPTBot` is different from blocking a search indexer like `OAI-SearchBot`, which can remove you from ChatGPT's search results. Decide per token, based on whether you want training, indexing, or live fetching, and measure the traffic first.

### Which AI bot crawls the most?

By crawler share, Googlebot still leads at roughly 27 percent, followed by GPTBot near 11.5 percent and Bytespider around 10.3 percent ([Cloudflare](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/), 2026). Grouped by company, OpenAI's bots account for about 69 percent of AI-driven traffic (HUMAN Security, 2026). Shares move monthly, so treat them as snapshots.

## Conclusion

The phrase "AI bot" hides four very different visitors. A crawler studying your docs, a fetcher closing a sale, an agentic browser pretending to be a person, and a search bot deciding whether to cite you are not the same event, and a dashboard that merges them tells you nothing useful. The taxonomy is the first real step toward managing this traffic instead of guessing at it.

You cannot govern what you cannot name. Start by classifying every request on the server, watch the labeled feed, and decide per type what you want to allow, block, or optimize for. Caprail sorts the traffic for you and grades what each agent received. It is free during the private beta, with no credit card required.

## Sources

- HUMAN Security, 2026 State of AI Traffic and Cyberthreat Benchmark, retrieved 2026-06-15, https://www.globenewswire.com/news-release/2026/04/09/3270682/0/en/human-security-s-2026-state-of-ai-traffic-cyberthreat-benchmark-report-signals-a-new-internet-era-automation-growth-now-outpaces-humans.html
- Cloudflare, From Googlebot to GPTBot: who's crawling your site, retrieved 2026-06-15, https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- Cloudflare, Radar 2025 Year in Review, retrieved 2026-06-15, https://blog.cloudflare.com/radar-2025-year-in-review/
- OpenAI, Bots and user agents documentation, retrieved 2026-06-15, https://developers.openai.com/api/docs/bots
- Anthropic, Does Anthropic crawl data from the web, retrieved 2026-06-15, https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler