# State of Docs: Agents are your top readers

**TL;DR:** Agents now make up more than half of the requests to the docs we host, and they read differently
  from people. They read more of the API reference and the changelog, and they almost always ask for
  Markdown. This post uses nine months of that traffic to show you how to build docs for this age.

In 2026 you have two readers, a human and a second one that never opens a tab. Here is where we think docs are going based on nine months of traffic across every docs site we host plus the available public data. We explain how we counted every number.

## By the numbers

- **6x**: More agents reading docs. Agent requests to Fern-hosted docs in September vs. January.
- **55%**: Agents outnumber people. Of large Fern-hosted docs sites now have more agent readers than human ones.
- **74%**: Agents read the reference. Of API reference and changelog page views come from agents.
- **93%**: Agents want Markdown. Of agent requests go to Markdown, not HTML.
- **92% vs. 17%**: Agents succeed with SDKs. Success rate of agent-built integrations with Fern's SDK vs. without it.

## Agents read differently

Your users read guides, but agents read the reference. Agents spend 31% of their reading on API reference, while people spend 18%.

The changelog makes the difference even clearer. Agents read it twice as much as people to understand what's changed on your platform. Agents also keep different hours. 21% of their requests arrive on weekends, compared with 16% of human page loads.

**Agents read more of the API reference and changelog**

|  | API reference | Changelog | Guides and other pages |
| --- | ---: | ---: | ---: |
| Agents | 31.2% | 3.4% | 65.4% |
| People | 17.5% | 1.3% | 81.2% |

*Source: Fern product analytics, September 2026. Sections inferred from URL paths*

**Agents read more on weekends than people do**

|  | Weekend | Weekday |
| --- | ---: | ---: |
| Agents | 21% | 79% |
| People | 16% | 84% |

*Source: Fern product analytics, September 2026*

**What to do:** Put the next round of work into the reference and the changelog, not just more guides.

1. Every parameter has a type, a description and an example.
2. Every error code says what causes it and how to fix it.
3. Every endpoint shows a working request and response.
4. The changelog lists each change by endpoint and date, so an agent can tell what's new.

**How Fern helps:** Fern Docs [generates the API reference](https://buildwithfern.com/learn/docs/api-references/generate-api-ref) from your OpenAPI spec, so every endpoint and parameter matches the API you ship. [Changelog pages](https://buildwithfern.com/learn/docs/configuration/changelogs) turn a folder of Markdown files into dated entries with search and tag filtering.

## Who's reading

"Agents" isn't one reader, and it helps to know which ones visit you. In September, Claude Code sent 1.03 million requests to Fern-hosted docs. That's 13% of all agent traffic and 3.4x the next named agent.

The classic AI crawlers, GPTBot and ClaudeBot, are far down the list. They read to train, and training reads a site once. A coding agent reads it every time a developer integrates. That means your docs are now read in developers' terminals, not just by search crawlers.

The developer behind the agent has changed too. In Stack Overflow's 2026 survey, search is still the first stop for answers, and asking an AI agent is now second. 64% of developers also visit Stack Overflow less for simple questions than they used to. That makes your docs the most important place to answer questions about your platform, for people and for the agents working for them. Developers also check where an answer came from. 93% say they need source attribution before they trust an AI answer.

**What to do:** Check your logs for the agents that name themselves in the user agent, and write for the coding agents at the top of that list. Then make your docs easy to cite. Give every page and heading a stable URL, so an agent that quotes your docs can link the exact section it used. If your docs have an AI assistant, make it cite its sources.

**How Fern helps:** The [Web Analytics tab](https://buildwithfern.com/learn/dashboard/metrics/web-analytics) in the Fern Dashboard breaks down AI bot traffic by provider, so you can see which agents read your docs without digging through server logs. [Ask Fern](https://buildwithfern.com/learn/docs/ai-features/ask-fern/overview) comes with every Fern Docs site and automatically cites the docs pages behind each answer.

## What we see on Fern-hosted docs

We pulled every request to every docs site we host from January 1 to September 30. Agent requests grew from 1.3 million a month to 7.9 million, while human page loads slipped from 5.9 million to 5.1 million. In September, agents made up 60.6% of requests across all sites. Your docs are probably following the same curve.

**Agent requests passed human page loads in 2026**

Monthly agent requests and human page loads across all Fern-hosted docs sites.

|  | Agent requests | Human page loads |
| --- | ---: | ---: |
| January 2026 | 1.3M | 5.9M |
| September 2026 | 7.9M | 5.1M |

*Source: Fern product analytics, all Fern-hosted docs sites, January and September 2026*

**The number to trust:** To keep new Fern sites from inflating the growth, we compare the same 90 sites in both months: every site with at least 1,000 human page loads in January and in September. On those same sites, agents went from 13.5% of requests to 51.7%. So the growth isn't just new sites.

**Traffic changes on the same 90 sites**

|  | Agent requests | Human page loads |
| --- | ---: | ---: |
| January 2026 | 13.5% | 86.5% |
| September 2026 | 51.7% | 48.3% |

*Source: Fern product analytics. The same 90 sites, each with 1,000+ human page loads in both January and September 2026*

## Agent share varies widely based on type of site

The 51.7% above combines traffic from all 90 sites, and individual sites look very different. Among the 89 Fern sites with at least 5,000 human page loads in September, agents made up less than 10% of traffic on some sites and more than 90% on others.

Agents make up the most traffic on docs for APIs that developers call from code, because coding agents read those docs while they write the integration. They make up the least on docs for products people use through a UI, where a person reads a page and then clicks through the product.

**What to do:** Compare the agent requests in your logs with your human page loads to see where your site falls. If agents are the larger share, prioritize the reference, the changelog and Markdown over new guides.

**How Fern helps:** [Web Analytics](https://buildwithfern.com/learn/dashboard/metrics/web-analytics) shows page views and AI bot traffic for your site in one place, so you can see where you fall without writing a log query. Fern then generates an agent-friendly API reference from your spec, and [Fern Agent automations](https://buildwithfern.com/learn/docs/ai-features/fern-agent#automations) draft changelog entries and Markdown guides on a schedule, each as a pull request for your team to review.

## Markdown won

Agents don't want your HTML, so stop making them parse it. HTML wraps every page in navigation, scripts and layout markup, and an agent pays tokens to read all of it. When Cloudflare launched Markdown for Agents, a feature that serves Markdown to agents, it measured its own announcement post at 16,180 tokens as HTML and 3,150 as Markdown. That's about 80% fewer tokens for the same content. On our sites, nearly every agent request now ends at a page's Markdown file.

**The same page costs about 5x fewer tokens as Markdown**

Tokens an agent spends to read one page, as HTML and as Markdown. Measured by Cloudflare.

| Format | Tokens |
| --- | ---: |
| HTML | 16,180 tokens |
| Markdown | 3,150 tokens |

*Source: Cloudflare, Introducing Markdown for Agents (Feb 12, 2026)*

What's still debated is how agents ask for it. Some fetch the `.md` URL, and others send an `Accept: text/markdown` header and let the server negotiate. Public testing shows both habits are common. Claude Code, Cursor and GitHub Copilot send the header, Codex CLI follows a Markdown link in the page head, and ChatGPT, Gemini and Perplexity fetch only HTML.

A header only helps agents that know to send it. A link to the `.md` file works for any agent that can follow a URL.

**What to do:** Serve every page as Markdown at a `.md` URL, and answer the `Accept: text/markdown` header too. Then link the `.md` file from `llms.txt`, from the top of the page's Markdown, and from the page head.

**How Fern helps:** [Every Fern page is available as Markdown](https://buildwithfern.com/learn/docs/ai-features/markdown) at its `.md` URL and through the `Accept: text/markdown` header, including API Reference endpoint pages. Fern also adds a [per-page directive](https://buildwithfern.com/learn/docs/ai-features/agent-directives) that points agents to the `.md` URLs and `llms.txt`.

## llms.txt won't bring you traffic, but it helps agents find your pages

`llms.txt` is the most argued-about file in docs. Google's John Mueller said in 2025 that no AI system was using it. A year later, Ahrefs checked 137,000 domains and found that 97% of valid `llms.txt` files got zero requests in May 2026.

On Fern sites, `llms.txt` was fetched 448,667 times in September. Among agents that name themselves, Claude Code asked for it most, with 32,812 requests. That is more than GPTBot, ClaudeBot and OAI-SearchBot combined. The file was designed with crawlers in mind, but it found its audience in coding agents.

Requests for `llms.txt` didn't drop. They rose from about 380,000 in January to 448,667 in September. What dropped was its share of agent requests, from 29% to 5.7%, because requests for the Markdown version of pages grew about 8x over the same months. That's how an index gets used: an agent reads `llms.txt` once, then fetches the Markdown for the pages it needs. It stops agents from wasting requests on wrong URLs, and that's reason enough to publish one.

**Markdown pages grew to 94% of agent requests as llms.txt shrank**

|  | January: Markdown pages | September: Markdown pages |
| --- | ---: | ---: |
| Agent requests | 71% | 94.3% |

*Source: Fern product analytics, January and September 2026*

We also stopped generating `llms-full.txt`. We learned quickly that one file holding the whole site was too large for most context windows, and agents didn't need it. `llms.txt` plus a Markdown version of each page gives them the same content, so `llms-full.txt` now redirects to `llms.txt`.

We learned the size limit on our own docs. When we ran Fern Agent Score on buildwithfern.com/learn, our root `llms.txt` failed its 50,000-character check, so we split it into product and version indexes.

**What to do:** Publish `llms.txt` as an index of your pages, split it by product and version once it grows large, and skip `llms-full.txt`. Then run your site through [Fern Agent Score](https://buildwithfern.com/agent-score), the free, open-source benchmark we built with Dachary Carey. It grades docs from 0 to 100 against the Agent-Friendly Docs spec, including whether your `llms.txt` exists, fits the size limit and can be found from any page.

**How Fern helps:** Fern [generates `llms.txt`](https://buildwithfern.com/learn/docs/ai-features/llms-txt) for every site and splits it by product and version, so it stays under the size limit as your docs grow. Every Fern Docs site also gets Markdown for every page and agent directives automatically, and our own docs at buildwithfern.com/learn score 99 out of 100.

## Every docs site needs an MCP server

MCP is almost two years old. The official registry opened in September 2025 with about 300 servers and was close to 2,000 by November. A hosted MCP server is now an expected docs feature, and every Fern site with Ask Fern ships one, so a developer can connect Claude Code or Cursor to the docs and search them without leaving their coding tool.

Serving answers is the easy part. Learning from what readers ask is harder, and there's more to learn from every month. Ask Fern, the assistant built into Fern docs, fielded 81,942 questions in January and 257,201 in September, more than 3x in nine months. Every question records what a developer or agent needed, and whether the docs had it.

Fern Agent Automations (now called Clerk) turns those questions into fixes. Since August it has edited pages and opened pull requests from the Dashboard, from Slack and from inside coding agents. Every change arrives as a pull request, so a person reviews it before it ships.

**What to do:** Review the questions readers ask your docs assistant each week, and fix the pages behind the ones it couldn't answer.

**How Fern helps:** Fern hosts an [MCP server](https://buildwithfern.com/learn/docs/ai-features/mcp-server) for every site with Ask Fern enabled. The [Fern Dashboard](https://buildwithfern.com/learn/dashboard/getting-started/overview) shows searches, Ask Fern conversations and [reader feedback](https://buildwithfern.com/learn/docs/user-feedback) in one place, and [Fern Agent](https://buildwithfern.com/learn/docs/ai-features/fern-agent) turns what you find into page edits and pull requests.

## WebMCP puts tools in the browser tab

Markdown and `llms.txt` help agents read your docs. MCP lets developers search them from an editor. WebMCP adds a third path: the page itself tells an agent in the browser what it can do. A site registers tools in JavaScript, such as "search the docs" or "fill in this form," and the browser hands them to the agent. The agent calls a typed tool instead of guessing which button to click, and it works inside the reader's logged-in session.

It isn't here yet. WebMCP is a W3C community group draft, not a standard. Chrome has tested it behind a flag since version 146 and as an origin trial since version 149, and in September the Chrome team asked to extend that trial to Chrome 162. Firefox and Safari haven't signaled support. It isn't on by default in any browser, and as of May, none of the major AI agents called WebMCP tools on live sites, according to one industry review. Google has said Gemini in Chrome will support it.

When it does arrive, three things change for docs:

- **It reaches agents that never ask for Markdown.** Agents that run in the browser read the page a person sees. With WebMCP, they could call a docs site's search or fetch a page's Markdown directly instead of scraping the HTML.
- **Gated docs and "try it" flows open up.** Tools run with the reader's session, so an agent could search private docs or make a test call with the reader's own API key, with no separate login. That also makes security the first question, because every tool acts as the user.
- **Tools need a source of truth too.** Hand-written tools drift from the API the same way hand-written SDKs do. Tools generated from the same definition as the docs stay in step.

For now, watch the Chrome and Edge release notes, and treat WebMCP as an experiment, not a requirement.

## One definition, every surface

An agent integrating your API doesn't only read. It also writes code that calls your SDK, your CLI or your MCP server. If those come from a different source than your docs, the agent reads one description of your API and calls another.

This is why we built Fern around a single API definition. When your docs and your SDKs are generated from the same source, they describe the same endpoints, types and errors, so the code an agent writes matches the API it read about.

We measured what that changes in our [SDK agent benchmark](/post/sdk-vs-raw-api-ai-agents-benchmark). Claude agents built the same Square payments integration on raw HTTP, with an SDK from OpenAPI Generator, and with Square's SDK, built with Fern. With Square's SDK, 11 of 12 integrations returned the right result and survived a rate-limit test, compared with 2 of 12 for each of the other setups. Agents using it also spent 75% to 85% fewer tokens fixing their own bugs than on raw HTTP, because the SDK already handled the retries and auth they otherwise had to write.

**What to do:** Generate your SDKs, CLI and MCP server from the same API definition as your docs, so agents read and call the same API.

**How Fern helps:** Fern generates [SDKs](/sdks) in 9 languages, a [CLI](/cli) and an [MCP server](https://buildwithfern.com/learn/mcp-generator/get-started/quickstart) from the same [API definition](https://buildwithfern.com/learn/api-definitions/overview/what-is-an-api-definition) as your [Docs](/docs).

## Ok, so where do I start?

You can start today without changing platforms. Work through these in order, since they're roughly sorted by payoff.

1. **Make the API reference complete.** Every parameter, every error and a working example for every endpoint.
2. **Keep a changelog agents can use.** List each change by endpoint and date.
3. **Serve Markdown for every page,** and put the `.md` link where an agent will see it.
4. **Publish `llms.txt` as an index, not a dump.** Split it by product and version. Skip `llms-full.txt`.
5. **Add a directive at the top of each page's Markdown** pointing to the Markdown and the index.
6. **Host an MCP server for your docs, and make the search behind it good.** Most sites have the server now. Good search is what sets yours apart.
7. **Score your site against an agent-readiness checklist.** Most sites fail on structure, not writing.
8. **Give agents a way to say "I couldn't find it,"** and route that signal to a person or an automation that opens a fix.
9. **Instrument agent traffic, and write down your definitions.**

Then measure what pageviews used to stand in for. Track steps from landing on the docs to a first successful API call, tokens per correct answer, wrong-page fetches per retrieval, share of answers with a citation, and agent share of requests (with the definition stated).

## What can Fern do for you?

We built Fern on one idea before any of this data existed. The API definition is the source of truth, and your docs, SDKs, CLI and MCP server should all be generated from it. Nine months of traffic data now back it up.

Every Fern site serves Markdown for every page with an agent directive on top, a hierarchical `llms.txt`, and Ask Fern, which answers with citations in the docs, in Slack and over an API. Each site also has a hosted MCP server and a standards-based API catalog, Fern Agent turns readers' questions into reviewed pull requests, and [Agent Score](https://buildwithfern.com/agent-score) shows any team what to fix. A storage rebuild this summer made our largest sites render about 6.4x faster, and since October [every Fern site is prebuilt as static pages](/post/static-docs). And when Cloudflare built Forge, it chose our generators for its SDKs.

## How we counted

We counted every request to a Fern-hosted docs site by calendar month in UTC. Agent requests are server-side requests for Markdown routes and `llms.txt`. Human page loads are browser page views from client-side analytics. Agent share is agent requests divided by agent requests plus human page loads.

We excluded our own cache refreshes, Agent Score checker, internal clients and uptime monitors. In September, that removed 17 million requests, which is more than double the agent count. That's why anyone publishing a number like this should say what they threw out. We also never count search crawlers like Googlebot as agents.

**Limits of this data:** Human page loads miss people with ad blockers, which raises agent share. Agents that fetch HTML instead of Markdown aren't counted, which lowers it. 68% of September's agent requests carry a browser or generic user agent, so for those requests "agent" rests on the route rule. That group includes scripts, curl and Python clients, agents that don't identify themselves, and some people who opened a Markdown page. If you count only requests whose user agent names an AI agent or crawler, agents still made up 32.8% of September requests, up from 2.3% in January.

## Sources

- Fern product analytics, Jan 1 to Sep 30, 2026, pulled Oct 16, 2026
- Cloudflare, Introducing Markdown for Agents, Feb 12, 2026; Introducing Forge, Sep 28, 2026; cloudflare/forge PR #17, Oct 8, 2026
- Stack Overflow, 2026 Developer Survey: Community and results announcement, Oct 6, 2026
- Ahrefs, We analyzed 137K sites: 97% of llms.txt files never get read, Jun 15, 2026; Search Engine Roundtable, Google says no AI system currently uses llms.txt, Jun 17, 2025
- acceptmarkdown.com agent support matrix, updated Jun 22, 2026
- Fern, [How we built Agent Score](/post/agent-score), Apr 7, 2026; [We tested AI agents with and without an SDK. The SDK won.](/post/sdk-vs-raw-api-ai-agents-benchmark), Oct 1, 2026
- Chrome for Developers, WebMCP and Join the WebMCP origin trial, Jun 9, 2026; blink-dev, Intent to Extend Experiment: WebMCP, Sep 28, 2026; W3C Web Machine Learning Community Group, WebMCP draft; StudioMeyer, WebMCP reality check, May 2026
- MCP maintainers, One year of MCP, Nov 25, 2025; Den Delimarsky, One year of MCP through a core maintainer lens, Nov 2025