# AI crawler share, Q2 2026: network data and our beta signal

_2026-06-15 · data, agents_

Across our first two weeks of beta traffic, OpenAI and Anthropic agents accounted for roughly two-thirds of every machine read of our pages. That is a sharper tilt than the network-wide picture, where Google still leads, and it is exactly the kind of difference first-party data exists to catch. This is a field note, not a benchmark. The sample is small and the window is short, so every Caprail number below is early signal, clearly labeled, set next to the network data that gives it context.

> **Key takeaways**
> - This is a field note, not a benchmark. Our first-party numbers come from 151 agent requests across two beta sites over 12 days (June 3 to 15, 2026). Read each as early signal.
> - Network-wide, AI crawlers are about 22 percent of bot traffic and bots are about 51 percent of all web traffic ([Cloudflare Radar](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/), 2026).
> - In our beta data, OpenAI and Anthropic agents made up about two-thirds of machine reads, a heavier tilt than Cloudflare's network mix.
> - AI referrals are real but tiny so far: 15 human visits, led by ChatGPT.
> - We cannot yet report the format we served agents (markdown or HTML). That instrumentation gap is the first thing we are fixing, and we are reporting it rather than hiding it.

## How was this measured?

Every figure labeled "our beta data" comes from Caprail's own server-side classification across two private-beta sites, for requests between June 3 and June 15, 2026. That is 2,857 total requests, of which 151 were classified as agents. Site identities are anonymized and the numbers are aggregated; we are reporting shape, not naming customers.

Two honest limits frame everything that follows. First, the sample is small. A 12-day window with 151 agent requests cannot support a confident benchmark, so we do not present one. Second, the window is short relative to how agents behave: crawlers front-load bulk reads over months, so a two-week slice catches them mid-stride. Where our numbers and the network numbers disagree, the network figures are the more reliable estimate of the steady state, and ours are the more honest estimate of what two real sites saw this fortnight. For the wider method, see [analytics for AI agents](/blog/analytics-for-ai-agents).

## What does AI traffic look like across the network?

Zoomed out, agents are now a structural share of the web, not a rounding error. As of 2026, Cloudflare Radar reports that bots account for about 51 percent of all web traffic and that AI crawlers are roughly 22 percent of that bot activity. Put together, agents are a double-digit slice of everything hitting a typical site, and the share is climbing.

| Measure (network-wide) | Value |
| --- | --- |
| Bots as a share of web traffic | ~51% |
| AI crawlers as a share of bot activity | ~22% |
| HTML requests coming from bots, not humans | 57.5% |

Source: [Cloudflare Radar](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/), 2026. Network-wide figures; any single site will differ.

The last row is the one that should change how you think about your dashboard. If well over half of HTML requests are bots, and your analytics only fires for browsers that run JavaScript, then your reporting is built on the minority of your traffic. That is the gap this whole report sits inside.

## Which vendors crawled us most?

In our beta window, OpenAI led machine reads, with Anthropic close behind and Google third. Counting only crawler and search-bot requests (the machine reads, setting aside live fetches and referrals), 111 requests split across vendors like this:

| Vendor | Machine reads | Share |
| --- | --- | --- |
| OpenAI | 44 | ~40% |
| Anthropic | 29 | ~26% |
| Google | 21 | ~19% |
| Microsoft | 6 | ~5% |
| Perplexity | 5 | ~5% |
| DeepSeek | 3 | ~3% |
| Others | 3 | ~3% |

Source: Caprail beta data, two sites, June 3 to 15, 2026. n = 111 machine reads. Counts this small move easily; read the order, not the decimals.

Now hold that next to the network. Cloudflare's May 2026 crawler share puts Googlebot well in front, with the OpenAI and Anthropic bots smaller:

| Crawler | Share of crawler traffic (network) |
| --- | --- |
| Googlebot | 27% |
| GPTBot | 11.5% |
| Bytespider | 10.3% |
| ClaudeBot | 9.7% |

Source: [Cloudflare Radar](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/), May 2026. Shares flip month to month; date-stamp them whenever you cite them.

The two tables disagree, and the disagreement is the point. Our sites skew toward an audience that builds with AI tooling, so OpenAI and Anthropic agents over-index on us relative to the open web. That is not noise to smooth away. It is the signal a site owner actually wants: not what the average site sees, but what yours does.

## What was the mix of crawlers, fetchers, and referrals?

It was more evenly split than we expected, with search bots narrowly ahead of crawlers and live fetchers a strong third. Of 151 agent requests, the breakdown by type was:

| Agent type | Requests | Share |
| --- | --- | --- |
| Search bots | 62 | 41% |
| Crawlers | 49 | 32% |
| Live fetchers | 39 | 26% |
| Agentic browsers | 1 | under 1% |

Source: Caprail beta data, June 3 to 15, 2026. n = 151 agent requests (out of 2,857 total; humans were the other 2,706).

The 39 live fetches stand out. Fetchers read a single page to answer a question someone is asking right now, so each one is closer to a human intent than a bulk crawl is. The named agents behind these requests were led by OpenAI's search bot and Anthropic's ClaudeBot and Claude-User, with Googlebot, GPTBot, and ChatGPT-User close behind. We even logged a single agentic-browser session, which a year ago would have been zero. For why fetchers matter most for conversion, see [the markdown twin](/blog/hello-agentic-web).

## Which assistants sent us people?

ChatGPT sent the most, but the whole referral pool is still tiny, so treat this as a first sighting rather than a trend. Across the window we recorded 15 human visits that an assistant referred:

| Assistant | Referred visits |
| --- | --- |
| ChatGPT | 10 |
| Perplexity | 3 |
| Claude | 2 |

Source: Caprail beta data, June 3 to 15, 2026. n = 15 referred human visits.

Fifteen visits is not a conversion study. What it does show is that referral attribution is working: these are people an assistant pointed at us, correctly separated from direct traffic that older tools would have lumped together. The volume will grow; the plumbing is already in place to count it when it does.

## How many times were we crawled per visitor sent?

In our window, roughly three crawler reads for every human an assistant referred. We logged 49 crawler requests and 15 AI-referred visits, a ratio near 3.3 to 1. That looks almost reasonable, until you set it beside the network.

| Source | Crawls per referral sent |
| --- | --- |
| Caprail beta (all crawlers, our two sites) | ~3.3 |
| ClaudeBot (network, derived) | ~11,000 to 24,000 |
| GPTBot (network, derived) | ~1,276 |
| Search (network, derived) | ~5 |

Sources: Caprail beta data, June 3 to 15, 2026 (n = 49 crawler requests). Network figures derived from Cloudflare Radar, 2026.

Our low ratio is not good news, it is a young-site artifact. Crawlers do their heavy bulk reading over months, and our sites are two weeks into a private beta, so the thousands-to-one passes simply have not happened yet. This is the metric we most expect to move by Q3, and it is exactly why a recurring report beats a one-off snapshot: the ratio only means something once you watch it change.

## Where did the agent traffic come from?

Mostly the United States, with a clear Austrian cluster that reflects where these beta sites are run from. Geography of the 151 agent requests:

| Country | Agent requests |
| --- | --- |
| United States | 114 |
| Austria | 25 |
| Unknown | 8 |
| Poland | 2 |
| Other | 2 |

Source: Caprail beta data, June 3 to 15, 2026. n = 151 agent requests.

The US share lines up with where the major agent infrastructure runs. The Austrian slice is the home-team effect of a small beta, and it is the kind of local skew that vanishes as a sample grows. We flag it rather than quietly dropping it.

## What we could not measure this quarter

The one metric we most wanted to publish, the format we actually served each agent, is not reportable for this window, and we would rather say so than fake it. Caprail is designed to grade every agent response as markdown or HTML, because agents read clean markdown far more reliably than browser markup. For this beta window, though, the served content type was recorded on only one request out of 2,857. That is an instrumentation gap on our side, not a finding about agents, and fixing it is the first item on the list before the next report.

We are calling this out on purpose. A data report's credibility rests on its methodology, and the fastest way to lose trust is to present a number you cannot stand behind. The markdown-versus-HTML grade is the metric only a server-side tool can produce, so it will headline the Q3 edition. This quarter it is honestly blank.

## What this means for site owners

The practical takeaway is that your vendor mix is probably not the network's vendor mix, and only first-party data will tell you yours. If your audience builds with a particular set of tools, the agents reading you will tilt the same way, as ours tilted toward OpenAI and Anthropic. That changes which crawlers are worth optimizing for and which assistants are worth courting.

Three moves follow from this report. Watch your own vendor share rather than assuming the published averages. Treat live fetchers as the high-intent segment, since each one maps to a real question. And get your crawl-to-referral ratio on record now, while it is small, so the trend is visible later. All three need server-side measurement, because a browser tool never sees the machine reads. For why that layer is the foundation, see [the best new analytics tool](/blog/the-best-new-analytics-tool).

## FAQ

### How was this report measured?

The first-party numbers come from Caprail's server-side request classification across two private-beta sites, for the window of June 3 to 15, 2026. That covers 2,857 total requests, 151 of them agents. Site identities are anonymized and figures are aggregated. Network figures are cited from Cloudflare Radar and labeled as such, never blended into our own counts.

### Why does your vendor share differ from Cloudflare's?

Because the audiences differ. Cloudflare measures a huge cross-section of the open web, where Googlebot leads. Our two beta sites attract people who build with AI tooling, so OpenAI and Anthropic agents over-index on us. Neither view is wrong; they answer different questions. The whole reason to measure your own traffic is that the average is not yours.

### What is a crawl-to-referral ratio?

It is how many times AI crawlers read your pages for each human an assistant sends back. A ratio of 3 to 1 means three machine reads per referred visitor. Network-wide, some crawlers run into the thousands per referral, while traditional search sits near 5 to 1. The ratio is only meaningful as a trend, watched over time on your own site.

### Is 151 requests enough to draw conclusions?

No, and we do not draw firm ones. A sample this small over 12 days is a field note: useful for direction and for proving the measurement works, not for benchmarking. That is why every first-party figure here carries its raw count, and why the headline framing is "what we saw," not "what is true of the web."

### When will the full benchmark report come?

When the data clears a responsible bar: enough sites, enough classified requests, and coverage across the major vendors. On current growth that points to a Q3 2026 edition, which will also restore the markdown-versus-HTML grade that this window could not support. We would rather delay a benchmark than publish a thin one.

## Conclusion

The network tells you the shape of AI traffic, and your own server tells you yours. This quarter the two diverged in a way worth noticing: across two beta sites, OpenAI and Anthropic agents read us more heavily than the open-web averages would predict, live fetchers showed up in real numbers, and the first AI referrals arrived. The sample is small and we have said so at every step, including where our own instrumentation fell short.

That is the trade a data report makes. Honest small numbers now, a clearer benchmark later, and a method you can check. Caprail is free during the private beta, and as more sites come online these numbers get sharper. Check back for the Q3 edition, when the ratio has had time to move and the format grade is back on the table.

## Sources

- Cloudflare, From Googlebot to GPTBot: who's crawling your site, retrieved 2026-06-15, https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- Caprail first-party beta data, two sites, June 3 to 15, 2026, server-side request classification (aggregated, anonymized)