Documentation that works for humans doesn't automatically work for the AI agents that developers rely on daily. Tools like GitHub Copilot and Cursor fetch your docs during active coding, but they receive HTML files bloated with markup that consumes 80-90% more tokens than the actual content requires. LLM-friendly docs serve markdown directly to these agents while keeping the full HTML experience for browsers. This approach immediately improves how accurately AI assistants reference your API specs, with benefits that compound across every documentation request throughout a development session.
TLDR:
- AI agents parse docs as raw data within token limits, making markdown 80-90% more efficient than HTML
llms.txtfiles create structured indexes AI tools find automatically atyoursite.com/llms.txt- Content negotiation serves markdown to AI agents while browsers get HTML, no configuration needed
- Complete API schemas with explicit types prevent AI hallucinations in generated code
- Fern auto-generates
llms.txtand serves markdown to AI agents while syncing docs from API definitions
How to write LLM-friendly documentation
LLM-friendly documentation serves both human developers and AI agents without sacrificing readability. AI coding assistants like Cursor, GitHub Copilot, and Claude retrieve docs directly during development. When these tools can't parse documentation properly, developers struggle to implement APIs correctly.
Successful implementations share key characteristics: structured markdown formatting, clear content hierarchies, machine-readable data formats, and version accuracy. These same principles improve the developer experience for human readers.
Creating docs that work for LLMs requires deliberate decisions about structure and formatting. The techniques below focus on the parsing, retrieval, and context extraction that AI agents need to assist developers properly.
Why AI agents read documentation differently than humans
AI agents retrieve documentation during active coding sessions, pulling content directly into their context windows to answer developer questions. Unlike humans who browse visually, AI tools parse raw content programmatically.
The retrieval process creates immediate constraints. AI agents work within token limits, treating each piece of documentation as consumable data instead of a browsable webpage. When an agent needs information about an API endpoint, it fetches the relevant page and converts it to a format it can process.
HTML creates serious token overhead. Converting documentation pages from HTML to markdown typically reduces token consumption by over 90%. This reduction directly impacts how much documentation an AI agent can reference within its context window.
JavaScript-heavy documentation sites present another barrier. AI agents cannot execute client-side code or interact with JavaScript-based navigation. They need direct access to content in structured, parsable formats.
Human readers scan headings and skim sections. AI agents need clean hierarchies and predictable structures to extract information quickly.
Implement llms.txt as your AI documentation index
Fern automatically generates and maintains both llms.txt and llms-full.txt files for your documentation site. These files update with every documentation build, creating a structured index that AI agents find at yoursite.com/llms.txt without manual configuration.
The llms.txt specification provides a curated entry point for AI tools seeking documentation content. Hundreds of thousands of websites have adopted this format, with adoption concentrated among developer tools, AI companies, and SaaS platforms.
Fern's llms-full.txt variant includes complete API schemas, resolved SDK code examples, and full markdown documentation in a single file. AI agents retrieve complete reference material in one request instead of following multiple links. You can filter llms-full.txt output with query parameters like lang and excludeSpec to return only Python-specific or TypeScript-specific examples, reducing token usage further.
Serve markdown directly to reduce token consumption
Fern serves markdown automatically to AI agents while delivering HTML to browsers, reducing token consumption by over 90% without any configuration. When coding agents like Cursor or Claude Code request documentation, Fern detects the request and returns raw markdown instead of HTML-wrapped content.
When AI coding assistants fit complete endpoint documentation in their context window instead of fragmented chunks, they generate correct implementation code on the first attempt. Agents retrieve parameter definitions, authentication requirements, and response schemas together instead of making multiple requests or truncating critical details.
Content negotiation requires no changes to how agents request documentation. Browser requests continue receiving the full HTML experience with navigation and styling. The server reads the incoming request and routes to the appropriate format automatically.
Control what AI agents see with visibility tags
Fern provides <llms-only> and <llms-ignore> tags that let you control what content AI agents and human readers see on each page.
Use <llms-only> to include content that only AI agents receive, such as verbose technical context, implementation details, or cross-references that help agents understand relationships between pages. This content stays hidden from human readers on your documentation site.
Use <llms-ignore> to hide content from AI agents while keeping it visible to humans, such as marketing CTAs, promotional callouts, or navigation hints that would waste tokens without adding useful context for an agent.
These tags affect all markdown output intended for AI consumption, including llms.txt endpoints, the Copy Page action, and the View as Markdown page action.
Structure content for retrieval augmented generation
RAG systems split documentation into chunks, index them, and retrieve relevant pieces when answering questions. The chunking strategy determines whether AI agents return complete, accurate information or fragmented responses.
Each section under a clear heading should contain a complete thought that makes sense when retrieved independently. Avoid spreading single concepts across multiple headings or burying critical details in long paragraphs.
Consistent heading hierarchies help AI agents understand document structure. Use H2 for major topics and H3 for subtopics throughout. Predictable patterns help retrieval systems identify which chunk contains the answer.
Explicit relationships matter more than implied connections. Instead of referencing "the authentication method mentioned earlier," state "OAuth 2.0 authentication" directly. Agents may retrieve each chunk without surrounding context.
This structure directly affects code accuracy. When an agent retrieves a complete endpoint explanation with parameters and authentication requirements in one chunk, it generates correct implementation code.
Optimize API reference documentation for machine parsing
Machine-readable API specifications like OpenAPI and AsyncAPI provide structured schemas that AI agents parse directly. These formats define endpoints, parameters, and response types in consistent JSON or YAML structures that retrieval systems process without ambiguity.
Complete type definitions prevent hallucination. When an API parameter accepts string, integer, or object, specify exactly which type and include constraints like format, minimum/maximum values, or required properties. Missing type information forces AI agents to guess, producing code that fails at runtime.
Parameter naming consistency across endpoints helps AI agents recognize patterns. If one endpoint uses user_id and another uses userId for the same concept, explain this explicitly in your documentation.
Error documentation needs complete schemas for every non-200 response code. AI agents generating error handling code need the exact fields that appear in error responses and their possible values.
According to a 2024 Gartner forecast, over 30% of the increase in API demand by 2026 would come from AI and LLM tools. Incomplete reference documentation produces incorrect integration code when developers rely on AI coding assistants.
Maintain documentation accuracy with version control
Fern generates API reference documentation directly from API definitions, making API reference drift structurally impossible. The API reference stays accurate because Fern creates it from OpenAPI or Fern Definition files instead of manually maintained pages. Code examples reflect actual SDK behavior because they come from the same source specification.
The CLI integrates into CI pipelines to catch breaking changes before they ship. Running fern diff in CI compares the local API definition against the latest version and detects changes that would break existing client code. Teams can configure these checks to block merges when APIs contradict published SDKs or documentation.
Git integration treats documentation as code. Teams edit markdown files in repositories, submit pull requests, and run validation in CI workflows. This prevents the lag between implementation and updated docs that creates accuracy gaps.
Stale documentation presents different risks for AI agents than for human developers. Humans apply skepticism when docs seem outdated. AI agents retrieve documentation on every request and treat it as truth, generating code based on deprecated endpoints or removed parameters.
Fern supports versioned documentation out of the box. Each API version gets accurate documentation accessible at version-specific URLs that AI agents can reference when developers work with older releases.
Generate AI-friendly documentation with Fern
Fern generates documentation from API definitions that works for both human developers and AI agents without extra configuration. The platform automatically creates llms.txt and llms-full.txt files with every documentation build, providing AI tools with a structured index at yoursite.com/llms.txt and complete API schemas with resolved SDK examples.
Content negotiation happens automatically. When AI coding assistants like Cursor or GitHub Copilot request documentation, Fern serves clean markdown instead of HTML, reducing token consumption by over 90%. Browsers continue receiving the full HTML experience with navigation and styling without any changes to how requests are made.
The API reference generated from OpenAPI or Fern Definition files remains machine-readable with complete type schemas, explicit parameter definitions, and full error response documentation. Every endpoint includes automatically generated code examples in Python, TypeScript, Go, Java, and other supported languages, giving AI agents accurate implementation patterns to reference.
Documentation stays synchronized with API changes through Git integration. Teams edit markdown files in repositories and run validation in CI workflows, preventing the lag between implementation and documentation updates. Version control treats documentation as code, and teams can configure CI checks to block merges when changes contradict published specs.
Developers can filter the API reference by language using query parameters on llms-full.txt, allowing AI tools to retrieve only Python- or TypeScript-specific documentation and reduce token usage. The platform includes "View as Markdown" and "Open in Claude/ChatGPT" page actions that let users load any documentation page directly into their AI tool.
Final thoughts on documentation that serves AI and humans
Creating LLM-friendly docs doesn't require choosing between human readability and machine parsing. Structured markdown and clear hierarchies that help AI agents retrieve information also make documentation easier for developers to scan and understand. Better structure benefits everyone reading your docs.
FAQ
How can I make my documentation work better with AI coding assistants?
Serve structured markdown directly to AI agents through content negotiation, implement llms.txt at your documentation root, and maintain consistent heading hierarchies that allow retrieval systems to extract complete chunks of information independently.
What is llms.txt and where should it live?
llms.txt is a markdown-formatted index file that lives at yoursite.com/llms.txt and provides AI agents with a structured entry point to your documentation, organizing links and descriptions in a format that coding tools like Cursor and GitHub Copilot can parse automatically.
Why does markdown reduce token consumption compared to HTML?
HTML documentation includes navigation menus, CSS classes, div wrappers, and script tags that AI agents cannot use, consuming up to 90% more tokens than the same content in plain markdown format, which directly limits how much documentation an agent can reference within its context window.
How does version control prevent AI agents from generating broken code?
Git-based documentation workflows tie documentation updates to API changes in the same pull request, preventing the lag between implementation and updated docs that causes AI agents to retrieve outdated endpoint information and generate code based on deprecated parameters.
Can API reference documentation be generated automatically to stay accurate?
API reference documentation generated from OpenAPI or Fern Definition files stays synchronized with actual API behavior because both the documentation and the code examples come from the same source specification, preventing the drift that occurs with manually maintained docs.
