> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://buildwithfern.com/learn/llms.txt.

## Custom robots.txt

seo, docs.yml

You can now serve your own [`robots.txt`](/learn/docs/seo/robots-txt) at the root of your documentation site by pointing `agents.robots-txt` at a file in your repo. Use this to opt in or out of specific AI crawlers like `GPTBot` or `ClaudeBot`, gate sensitive sections from indexing, or signal training and search preferences with the [Cloudflare Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/). Your file is served verbatim at `/robots.txt`, and Fern appends a managed block disallowing internal API routes after your content.

**`docs.yml`**

```yaml docs.yml
agents:
  robots-txt: ./robots.txt
```

Read the docs

## Multi-source docs

customization, docs.yml

Multi-source docs lets each team publish independently to a shared custom domain — for example, [docs.nvidia.com](https://docs.nvidia.com) spans sub-paths like `/nvcf`, `/brev`, and `/aiperf`, each owned by a different product team and repository.

Set `multi-source: true` on the instance and reference a [global theme](/learn/docs/customization/global-themes) for consistent branding across repositories.

**`docs.yml`**

```yaml docs.yml
global-theme: my-org-theme

instances:
  - url: example.docs.buildwithfern.com/product-a
    custom-domain: docs.example.com/product-a
    multi-source: true
```

Read the docs