> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://buildwithfern.com/learn/llms.txt.

# Index Website

POST https://fai.buildwithfern.com/sources/website/{domain}/index
Content-Type: application/json

Start crawling and indexing a website.
Returns a job_id to track the crawling progress.

Reference: https://buildwithfern.com/learn/docs/ai-features/ask-fern/api-reference/website/index-website

## Authentication

- `Authorization` header (bearer token, required) — Bearer authentication of the form `Bearer <token>`, where token is your auth token.

## Servers

- `https://fai.buildwithfern.com` (Production, default)
- `https://fai-dev.buildwithfern.com` (Development)
- `http://localhost:8080` (Local)

## Request

### Path parameters

- `domain` (string, required)

### Body (application/json)

This endpoint expects an object.

- `base_url` (string, required) — The base URL to start indexing from (e.g., 'https://docs.example.com')
- `domain_filter` (string, optional, nullable) — Domain to filter crawling (e.g., 'docs.example.com'). Defaults to base_url domain.
- `path_filter` (string, optional, nullable) — Path prefix to restrict crawling (e.g., '/docs'). Only URLs starting with this will be crawled.
- `url_pattern` (string, optional, nullable) — Regex pattern to filter URLs (e.g., `https://example\.com/(docs|api)/.*`).
- `chunk_size` (integer, optional, nullable, default: 1000) — Size of text chunks for splitting documents
- `chunk_overlap` (integer, optional, nullable, default: 200) — Overlap between consecutive chunks
- `min_content_length` (integer, optional, nullable, default: 100) — Minimum content length to index a page
- `max_pages` (integer, optional, nullable) — Maximum number of pages to crawl. None means unlimited.
- `delay` (double, optional, nullable, default: 1) — Delay in seconds between requests
- `version` (string, optional, nullable) — Version to tag all indexed pages with
- `product` (string, optional, nullable) — Product to tag all indexed pages with
- `authed` (boolean, optional, nullable) — Whether indexed pages should be auth-gated

## Response

### 200

Successful Response

- `job_id` (string, required) — ID to track the indexing job status
- `base_url` (string, required) — The base URL being indexed

## Errors

### 422 Unprocessable Entity Error

Validation Error

- `detail` (list of object, optional)
  - `loc` (list of string or integer, required)
  - `msg` (string, required)
  - `type` (string, required)

## Examples

**Request**

```json
{
  "base_url": "string"
}
```

**Response**

```json
{
  "job_id": "string",
  "base_url": "string"
}
```