AI featuresAsk Fern

Additional content sources

View as Markdown

Extend Ask Fern’s knowledge beyond your core documentation by adding additional content sources like internal FAQs, support tickets, blog posts, and knowledge base articles.

Publicly available content

For content that’s publicly accessible on the web — like marketing sites, blog posts, or external knowledge bases — Ask Fern can automatically crawl and index it. There are two ways to set this up:

docs.yml configuration

The simplest approach is to add URLs directly in your docs.yml configuration under ai-search.datasources:

docs.yml
1ai-search:
2  datasources:
3    - url: https://example.com/additional-docs
4      title: Additional documentation
5    - url: https://blog.example.com
6      title: Company blog

Each datasource requires a url and accepts an optional title that helps users understand where cited content comes from.

Websites API

For more control over what gets crawled, use the Websites API. This lets you apply filters to target specific subdomains or URL paths:

FilterDescriptionExample
domain_filterRestrict crawling to a specific subdomainhelp.example.com will only crawl pages on that subdomain, not www.example.com or docs.example.com
path_filterRestrict crawling to a specific section of the site/getting-started will only crawl URLs containing /getting-started in the path, like docs.example.com/getting-started, but not docs.example.com/api-reference

Here’s an example using path_filter:

Example
1{
2    "base_url": "https://docs.example.com",
3    "path_filter": "/getting-started"
4}

The API returns a job_id to track the crawling progress. When referenced, Ask Fern cites the original URL where the content was found.

Not publicly available content

For content that isn’t publicly accessible, like internal documentation, support ticket summaries, or proprietary knowledge base articles, use the Documents API to upload markdown content directly. This gives you precise control over what gets indexed.

Example
1{
2    "document": "Ferns are plants native to the tropical and subtropical regions of the world. They are characterized by their fronds, which are large, leaf-like structures that are often found in the understory of forests.",
3    "title": "What are ferns?",
4    "url": "https://en.wikipedia.org/wiki/Fern"
5}

The URL is used solely for citations — Ask Fern doesn’t crawl it. You provide the full content in the document field.