# The markdown twin: how to serve clean content to AI agents

_2026-06-11 · agents, markdown_

More and more of the traffic on the web is not people. AI agents read pages, follow links, and answer questions on behalf of the people who used to click through themselves. Most sites still serve those agents the same heavy HTML they serve a browser, and the agents spend their budget parsing layout and scripts instead of reading content. This post explains the fix this site uses: a markdown twin of every page, why clean markdown reaches agents that HTML cannot, and how to add one to your own site.

> **Key takeaways**
> - A markdown twin is the same page served as clean markdown when an agent asks for it. Converting one real page from HTML to markdown cut token use by about 80 percent ([Cloudflare](https://blog.cloudflare.com/markdown-for-agents/), 2026).
> - Major AI crawlers fetch JavaScript files but never run them, so client-rendered content is invisible to them ([Vercel and MERJ](https://vercel.com/blog/the-rise-of-the-ai-crawler), 2024).
> - AI crawlers were about 20 percent of verified bot traffic in 2025, and GPTBot's share more than doubled year over year ([Cloudflare Radar](https://blog.cloudflare.com/radar-2025-year-in-review/), 2025).
> - One MDX source produces both the page a person reads and the markdown an agent reads, so there is no second copy to keep in sync.

## What is a markdown twin?

A markdown twin is the same page, served as clean markdown when an agent asks for it instead of as browser HTML. Append `.md` to any path on this site, or send an `Accept: text/markdown` header, and you get the content with none of the layout, navigation, or scripts wrapped around it. The payoff is concrete. When Cloudflare converted one of its own blog posts from HTML to markdown, the token count dropped from 16,180 to 3,150, about an 80 percent reduction ([Cloudflare](https://blog.cloudflare.com/markdown-for-agents/), 2026).

Why does that matter so much? Because an agent pays for every token it reads, in money and in context window. Most of the tokens in a modern web page are not the words a reader cares about. They are the div soup, inline styles, tracking tags, and framework runtime that a browser needs and a language model does not. Strip that away and you hand the agent the part it actually wanted.

This is not a fringe idea anymore. In February 2026 Cloudflare shipped markdown content negotiation across its network using the same `Accept: text/markdown` mechanism described here ([Cloudflare](https://blog.cloudflare.com/markdown-for-agents/), 2026). When the infrastructure layer of the web starts serving markdown to agents, a per-page twin stops being a clever trick and starts being table stakes.

## Why HTML fails AI agents

HTML fails agents because most of them never run it the way a browser does. In a study of roughly 1.3 billion AI crawler fetches in a single month, the major crawlers fetched JavaScript files but executed none of them, so any content rendered on the client was invisible to them ([Vercel and MERJ](https://vercel.com/blog/the-rise-of-the-ai-crawler), 2024). GPTBot requested JavaScript on about 11.5 percent of fetches and ClaudeBot on about 23.8 percent, then ran zero of it. Only Google and Apple's bots rendered pages at all.

The contrast is stark when you line the two up:

| What the agent gets | Heavy HTML page | Markdown twin |
| --- | --- | --- |
| Tokens for one real page | 16,180 | 3,150 |
| Client-rendered content | Invisible, no JavaScript run | Served in full |
| Effort spent on layout and scripts | High | None |

Source: token figures from Cloudflare, 2026; JavaScript execution from Vercel and MERJ, 2024.

Think about what that means for a site built on a client-rendered framework. The browser downloads a thin HTML shell, runs the JavaScript, and paints the real content. A person sees the finished page. An agent sees the shell. Your most important text might as well not exist, because the visitor that was supposed to read it does not have a renderer.

And these are not rare visitors. AI crawlers made up about 20 percent of verified bot traffic in 2025, with GPTBot's share rising from 4.7 percent in mid 2024 to 11.7 percent a year later ([Cloudflare Radar](https://blog.cloudflare.com/radar-2025-year-in-review/), 2025). The audience reading you without a browser is large and growing fast. Serving it a shell is a quiet, expensive miss, and traditional analytics will not even show you it happened. For why that blind spot exists, see [why traditional analytics can't see AI agents](/blog/why-traditional-analytics-cant-see-ai-agents).

## How the markdown twin works on this site

The twin works by deriving the markdown from the same source the page is built from, then choosing the format per request on the server. Every public page here has a twin, and there is never a second file to keep current. The build step scans each page, derives the markdown version, and registers it in one manifest. The middleware then reads the request and serves the right format before any browser script could run.

The decision the middleware makes is small and boring on purpose:

```ts
// Simplified: the decision the middleware makes on every request.
function clientWantsMarkdown(req: Request): boolean {
  if (req.method !== "GET" && req.method !== "HEAD") return false;
  return prefersMarkdown(req.headers.get("accept"));
}
```

That is the whole gate. A normal browser sends an `Accept` header that prefers HTML, so it gets the page. An agent that asks for `text/markdown`, or that requests the `.md` path, gets the clean text. We built it on the server for one reason: the server is the only layer every visitor passes through, human or agent, before a single line of JavaScript executes. A browser tool sees the people who run scripts. Middleware sees everyone.

The result holds up in practice. The same content reaches a person as a rendered page and an agent as plain markdown, from one source, with no drift between the two. That single-source property is the part that makes this maintainable rather than a chore.

## This blog is its own example

This blog runs on the exact mechanism the rest of the site uses, so it doubles as a working demo. Each post is a single MDX file. That one file produces the page you are reading now and the markdown twin an agent reads. There is no parallel markdown copy authored by hand, and no export step that can fall out of date.

Want to see it? Fetch this post as markdown:

```sh
curl -H "Accept: text/markdown" https://caprail.dev/blog/hello-agentic-web
```

If you are reading this through an agent right now, you already did. The page rendered in a browser and the text returned by that curl come from the same file, which is the point. Dual-rendering from one source is what keeps the twin honest over hundreds of posts.

## Giving agents an index: llms.txt

A twin solves a single page, but an agent still has to find your pages. That is what an index file is for. The convention here follows the llms.txt proposal, a markdown file that lists a site's high-value content so a model can assemble context without crawling blindly ([Answer.AI](https://www.answer.ai/posts/2024-09-03-llmstxt.html), 2024). It was proposed in September 2024 and has since been picked up across developer-facing tools and documentation sites ([Search Engine Land](https://searchengineland.com/llms-txt-proposed-standard-453676), 2025).

This site serves two of them. `/llms.txt` lists every page with a one line summary, so an agent can scan the map first. `/llms-full.txt` includes the full markdown bodies, so an agent that wants everything can take it in one request rather than fetching a hundred pages. One is the table of contents, the other is the whole book. Together with per-page twins, they give an agent three clean ways in: the index, the bulk file, or any single page on demand.

## How to add a markdown twin to your site

The short version is to render twice from one source and choose the format on the server, never in the browser. You need three pieces: a way to derive markdown from each page's source, content negotiation that reads the `Accept` header or the `.md` suffix, and an index file or two so agents can discover what exists. Keep the derivation tied to the original source so the twin can never drift from the page.

There's a measurement gap most teams hit next. Once you serve both formats, which one did each agent actually receive? You cannot improve what you cannot see, and a browser tool sees none of this. Caprail classifies every request on the server and grades the response by format, markdown or HTML, per agent. Adding it is two lines in your middleware:

```ts
import { createCaprailMiddleware } from "@caprail-dev/analytics/next";

export const middleware = createCaprailMiddleware();
```

From there you get a live feed of agent requests and a per-page view of which engines read you and what they were served. A page that agents fetch often but always receive as heavy HTML is a page leaving value on the table, and now you can find it. For the wider picture of what server-side agent analytics covers, see [analytics for AI agents](/blog/analytics-for-ai-agents) and [the best new analytics tool](/blog/the-best-new-analytics-tool).

## FAQ

### What is a markdown twin?

A markdown twin is the same web page served as clean markdown when an agent requests it, instead of browser HTML. It strips the layout, scripts, and tracking that a model does not need. Converting one real page this way cut token use by about 80 percent ([Cloudflare](https://blog.cloudflare.com/markdown-for-agents/), 2026), which saves the agent both money and context.

### Do AI agents really not execute JavaScript?

Most do not. Across roughly 1.3 billion crawler fetches, the major AI crawlers downloaded JavaScript files but ran none of them ([Vercel and MERJ](https://vercel.com/blog/the-rise-of-the-ai-crawler), 2024). Only Google and Apple's bots render pages. So any content drawn on the client after load is invisible to them, which is why server-rendered clean text matters.

### What is the difference between a markdown twin and llms.txt?

A markdown twin is a clean version of one page, served on demand. An llms.txt file is an index that lists your high-value pages so an agent can find them ([Answer.AI](https://www.answer.ai/posts/2024-09-03-llmstxt.html), 2024). One is the content, the other is the map. You want both: the index so agents discover your pages, and twins so each page reads cleanly.

### Will serving markdown to agents hurt my SEO or my human visitors?

No. A person's browser sends an `Accept` header that prefers HTML, so it still gets the normal rendered page. Only a client asking for markdown, or the `.md` path, receives the twin. The human experience and your search markup are untouched. You are adding a second representation, not replacing the first.

### How do I know whether agents got markdown or HTML?

You measure it on the server, the one layer every request passes through. Browser analytics cannot see most agents at all, since AI crawlers were about 20 percent of verified bot traffic in 2025 ([Cloudflare Radar](https://blog.cloudflare.com/radar-2025-year-in-review/), 2025) and run no scripts. A server-side tool grades each response by format per agent, so you can spot pages served as heavy HTML.

## Conclusion

The web is gaining a second audience that does not use a browser, and it is already a fifth of verified bot traffic. Those readers pay for every token, skip your JavaScript, and want the content without the chrome. A markdown twin gives it to them from the same source that serves your human page, an index file helps them find it, and server-side measurement tells you whether it is working. None of it runs in a visitor's browser, and none of it slows your pages down.

Start small. Serve one page two ways, publish an llms.txt, and watch what the agents do. Caprail is free during the private beta, with no credit card required, and most sites find an agent they never knew was reading them within the first hour. To see how to attribute the humans those agents send back, read [how to track AI referral traffic](/blog/how-to-track-ai-referral-traffic).

## Sources

- Cloudflare, Introducing Markdown for Agents, retrieved 2026-06-15, https://blog.cloudflare.com/markdown-for-agents/
- Cloudflare Radar, 2025 Year in Review, retrieved 2026-06-15, https://blog.cloudflare.com/radar-2025-year-in-review/
- Vercel and MERJ, The rise of the AI crawler, retrieved 2026-06-15, https://vercel.com/blog/the-rise-of-the-ai-crawler
- Answer.AI, The /llms.txt file proposal, retrieved 2026-06-15, https://www.answer.ai/posts/2024-09-03-llmstxt.html
- Search Engine Land, Meet llms.txt, a proposed standard for AI website content, retrieved 2026-06-15, https://searchengineland.com/llms-txt-proposed-standard-453676