# API Docs for AI Agents: llms.txt Guide May 2026 | Fern

**Date:** 2026-02-16  
**Author:** Nathan Lian  
**Source:** https://buildwithfern.com/post/optimizing-api-docs-ai-agents-llms-txt-guide

> Learn how to optimize API documentation for AI agents using llms.txt in May 2026. Complete guide to llms.txt implementation for better AI code generation.

---
AI coding assistants like Cursor and GitHub Copilot need to read your API documentation to generate working integration code. The problem? These tools struggle to extract useful information from HTML documentation pages efficiently. They hit token limits parsing navigation menus, JavaScript, and visual styling before reaching your endpoint descriptions. When your [API documentation](https://buildwithfern.com/) isn't accessible to AI tools, developers get incorrect code suggestions and slower integration. The `llms.txt` format solves this by providing a markdown-based version of your docs that AI systems can parse within their token budgets. It strips away everything except the information needed for code generation.

**TLDR:**

-   `llms.txt` provides a compact overview with links to full documentation, while `llms-full.txt` embeds complete content directly in the file without requiring external fetches
-   AI agents need explicit parameter descriptions and error schemas to generate working code
-   Include only technical reference material — strip marketing copy, decorative elements, and anything that doesn't help with code generation
-   Tools like Fern auto-generate both `llms.txt` and `llms-full.txt` from your documentation as part of the build process

## What is `llms.txt` and why does it matter for API documentation?

`llms.txt` is a plain-text, markdown-based format designed for AI consumption. It serves a similar role to a sitemap: instead of helping search engines crawl your site, it helps coding assistants locate and understand your API's endpoints, authentication methods, and response schemas.

The format strips away CSS, navigation elements, and JavaScript, leaving clean structured text that AI systems can parse efficiently within their token budgets. In the context of API documentation, coding assistants can spend their context window on your endpoint descriptions, parameter types, and error schemas instead of page chrome.

With [85% of developers](https://www.ideaplan.io/blog/ai-coding-assistant-market-share-2026) using AI tools in 2026, writing integration code happens inside assistants like Cursor and GitHub Copilot. If your docs aren't accessible to these agents, you add friction to developer onboarding. `llms.txt` removes that friction.

## Understanding the llms.txt specification and format

The `llms.txt` specification uses a structured markdown format optimized for machine readability. Each file starts with an H1 header containing your project name, followed by a blockquote summarizing your API in one or two sentences.

Documentation is organized into sections using H2 headers for major topics like authentication or API endpoints. Within each section, provide core details in plain markdown with links to the corresponding full documentation pages. These links allow AI assistants to reference the condensed `llms.txt` content while following URLs for deeper context when needed.

Keep descriptions focused on what developers need to integrate your API: parameter types, authentication requirements, and error handling. Avoid marketing language or context that fails to serve code generation. Use markdown code blocks for examples that show usage patterns and clarify how your API behaves in practice.

## llms.txt vs. llms-full.txt: choosing the right approach

The specification defines two file variants that serve different AI consumption patterns. `llms.txt` provides a compact overview with links to full documentation. `llms-full.txt` embeds complete content directly in the file without requiring external fetches.

`llms.txt` works best for larger documentation sites where including everything would exceed token limits. AI assistants read the summary, identify relevant sections, and follow links for detailed information.

`llms-full.txt` suits APIs with concise documentation that fits entirely within AI context windows. The file contains complete endpoint descriptions, authentication details, and code examples without requiring link traversal.

Most API providers should implement both files. Serve `llms.txt` for AI assistants optimizing for token usage and `llms-full.txt` for agents that prefer self-contained context. Tools like Fern generate both variants automatically from the same source specification, eliminating manual maintenance.

|  | `llms.txt` | `llms-full.txt` |
| --- | --- | --- |
| Content | Summary with links to full docs | Complete content embedded |
| Best for | Large documentation sites | Concise APIs that fit in context windows |
| Requires external fetches | Yes | No |
| Token usage | Lower | Higher |

## Best practices for generating and maintaining llms.txt files

Keep your `llms.txt` accurate and current — stale files generate broken code. Outdated authentication flows or removed endpoints cause AI tools to suggest non-functional code, so breaking changes need immediate regeneration. Minor updates like typo fixes, clarified descriptions, and small parameter changes can wait for scheduled regeneration cycles.

Test AI tool parsing by loading your `llms.txt` file into Cursor or Claude and asking specific questions about your API. Request code for authenticating with your API, calling a specific endpoint, or handling a particular error response. If the AI tool references outdated information or can't locate documented features, adjust your file structure. Token count matters less than whether AI assistants can extract correct information quickly.

Monitor which sections AI tools reference most frequently by tracking link clicks from `llms.txt` to your full documentation pages. High-traffic sections deserve more detail in the `llms.txt` summary, while rarely-accessed content can rely on brief descriptions with links. This usage data reveals whether your information hierarchy matches how developers actually integrate your API. Tools like Fern provide analytics for `llms.txt` usage, including traffic breakdowns by LLM provider and page-level metrics showing bot versus human visitors.

API versioning adds complexity to `llms.txt` maintenance. When releasing a new API version, you can either maintain separate `llms.txt` files per version or include all versions in a single file. Separate files prevent token bloat but require AI tools to know which version a developer is targeting. A unified file with version-specific sections works better when most endpoints remain stable across versions. If your documentation tooling supports versioned docs, generating separate `llms.txt` files per version gives you the most control over what each AI assistant sees.

## Optimizing OpenAPI specifications for AI agent discovery

OpenAPI specifications serve as machine-readable contracts that let [AI agents](https://composio.dev/content/apis-ai-agents-integration-patterns) directly identify endpoints, methods, parameters, and response schemas. While `llms.txt` helps with general documentation discovery, a well-structured OpenAPI file gives agents structured access to your API's capabilities without relying on natural language interpretation.

Be thorough with every parameter — what seems obvious to a human reader is ambiguous to an AI agent. Include realistic example responses in your OpenAPI schema. AI coding assistants use these examples to generate response parsing code. Generic placeholder values like `"string"` or `123` produce less accurate code suggestions than responses that reflect your actual API output. Some documentation tools, including Fern, can [generate realistic examples automatically](https://buildwithfern.com/learn/docs/ai-features/ai-examples), but even manually written examples substantially improve code generation accuracy.

Document all error responses with status codes and schemas. When AI tools know your API returns a 429 with retry-after headers, they can generate proper error handling code that respects rate limits. Error schemas are easy to overlook, but they're some of the highest-value content for AI-assisted code generation.

## How Fern automatically generates and optimizes llms.txt for your API docs

Fern generates [automated llms.txt files](https://buildwithfern.com/learn/docs/ai-features/llms-txt) (both `llms.txt` and `llms-full.txt` variants) when building documentation sites. Fern also serves clean Markdown instead of HTML on any page URL — agents can fetch the source by appending `.md` to the URL or sending an `Accept: text/markdown` header. Together, these features reduce token consumption by over 90% compared to AI tools parsing full HTML pages.

The generation process pulls from the same API specification that creates your SDK code and API reference. When an endpoint or authentication method is updated in your OpenAPI file or Fern Definition, the `llms.txt` files update automatically during the next documentation build. This eliminates drift between your human-readable docs and AI-optimized files.

Both files are available at any level of your documentation hierarchy, not limited to the site root. You can access `/docs/llms.txt`, `/docs/ai-features/llms-full.txt`, or any other section path, letting AI tools request only the portion of your documentation relevant to a specific task.

Query parameters provide additional control over AI-accessible content. Append `?lang=python` to filter code examples to a specific SDK language, or use `?excludeSpec=true` to remove OpenAPI and AsyncAPI specifications and focus AI tools on conceptual guides instead of raw schemas. Fern automatically serves your OpenAPI and AsyncAPI specs at `/openapi.json` and `/asyncapi.json` (with YAML variants also available), linking them directly in `llms.txt`. Fern docs sites also expose a standards-based API catalog at `/.well-known/api-catalog` (per [RFC 9727](https://www.rfc-editor.org/rfc/rfc9727)), letting AI agents and MCP clients discover your APIs without scraping HTML. Fern also automatically prepends a default agent directive to every page served to AI agents, pointing them to your `.md` URLs, `llms.txt`, and `llms-full.txt` — you can override or disable this directive via the `agents` key in `docs.yml`. For custom `llms.txt` files, point to your files under the same `agents` configuration to tailor exactly what Cursor and GitHub Copilot see.

Custom markdown tags control content visibility between human readers and AI agents. Wrap content in `<llms-only>` tags to include information exclusively in `llms.txt` files. This works well for adding technical context like implementation details or architecture notes that would clutter the visual documentation. Use `<llms-ignore>` tags to exclude marketing CTAs, promotional content, or navigation hints that add no value for AI parsing. To exclude entire pages from `llms.txt` output, add `noindex: true` to the page's frontmatter. This removes the page from both AI-optimized files and site navigation while keeping it accessible by direct URL.

## Final thoughts on `llms.txt` adoption for APIs

Developers are already writing integration code inside AI coding assistants. `llms.txt` provides those tools with clean, structured access to your API documentation instead of struggling to parse HTML pages. When `llms.txt` is [generated automatically from your API specification](https://buildwithfern.com/), your AI-optimized documentation stays current with every API change, and developers get accurate code suggestions on their first try instead of dealing with hallucinated endpoints.

You can also use Fern's [Agent Score tool](https://buildwithfern.com/post/agent-score) to measure how well your documentation serves autonomous agents and see where you stand on the public leaderboard.

Want to see automated `llms.txt` generation in action? [Book a demo](https://buildwithfern.com/book-demo) with the Fern team.

## FAQ

### How do you generate an `llms.txt` file for API documentation?

The most reliable approach is to integrate `llms.txt` generation into your documentation build pipeline instead of writing the file manually. If you use Fern, `llms.txt` and `llms-full.txt` files are generated automatically from your documentation when building your site, typically reducing token consumption by over 90% compared to HTML pages.

### What's the difference between `llms.txt` and `llms-full.txt`?

`llms.txt` provides a compact overview with links to full documentation, making it ideal for larger documentation sites where including everything would exceed token limits. `llms-full.txt` embeds complete content directly in the file without requiring external fetches, which works best for APIs with concise documentation that fits entirely within AI context windows.

### When should you use MCP instead of `llms.txt` for an API?

Use `llms.txt` when helping developers write better integration code faster through AI-assisted tools like Cursor and GitHub Copilot. Choose MCP when AI agents should interact with your API autonomously without a human developer in the loop. MCP traditionally requires running your own protocol server, though some platforms — Fern, for example — [auto-host an MCP server](https://buildwithfern.com/learn/docs/ai-features/mcp-server) for your docs site so you skip the infrastructure entirely.

### How often should you update your `llms.txt` file?

Update your `llms.txt` file whenever you publish documentation changes, and integrate generation into your build pipeline so updates happen automatically. AI coding assistants cache these files, so stale versions can persist for days or weeks, leading to incorrect code suggestions if your documentation has changed.

### What information should be included in an OpenAPI specification for AI agents?

Include explicit parameter descriptions stating format requirements and validation rules, realistic example responses instead of generic placeholders, and all error responses with status codes and schemas. AI agents need this level of detail to generate accurate code. Unlike human developers, they cannot infer missing information from naming conventions or context.