Additional content sources

Extend Ask Fern’s knowledge beyond your core documentation by adding additional content sources like internal FAQs, support tickets, blog posts, and knowledge base articles. When Ask Fern references content from custom sources, it includes the associated URL as a citation.

Publicly available content

For content that’s publicly accessible on the web — like marketing sites, blog posts, or external knowledge bases — Ask Fern can automatically crawl and index it. There are two ways to set this up:

docs.yml configuration

The simplest approach is to add URLs directly in your docs.yml configuration under ai-search.datasources:

1
ai-search:
2
  datasources:
3
    - url: https://example.com/additional-docs
4
      title: Additional documentation
5
    - url: https://blog.example.com
6
      title: Company blog

Each datasource requires a url and accepts an optional title that helps users understand where cited content comes from.

Websites API

For more control over what gets crawled, use the Websites API. This lets you apply filters to target specific subdomains or URL paths:

Here’s an example using path_filter:

1
{
2
    "base_url": "https://docs.example.com",
3
    "path_filter": "/getting-started"
4
}

The API returns a job_id to track the crawling progress. When referenced, Ask Fern cites the original URL where the content was found.

Not publicly available content

For content that isn’t publicly accessible, like internal documentation, support ticket summaries, or proprietary knowledge base articles, use the Documents API to upload markdown content directly. This gives you precise control over what gets indexed.

1
{
2
    "document": "Ferns are plants native to the tropical and subtropical regions of the world. They are characterized by their fronds, which are large, leaf-like structures that are often found in the understory of forests.",
3
    "title": "What are ferns?",
4
    "url": "https://en.wikipedia.org/wiki/Fern"
5
}

The URL is used solely for citations — Ask Fern doesn’t crawl it. You provide the full content in the document field.