# Index Website

POST https://fai.buildwithfern.com/sources/website/{domain}/index
Content-Type: application/json

Start crawling and indexing a website.
Returns a job_id to track the crawling progress.

Reference: https://buildwithfern.com/learn/docs/ai-features/ask-fern/api-reference/website/index-website

## OpenAPI Specification

```yaml
openapi: 3.1.1
info:
  title: Index Website
  version: endpoint_website.index_website
paths:
  /sources/website/{domain}/index:
    post:
      operationId: index-website
      summary: Index Website
      description: |-
        Start crawling and indexing a website.
        Returns a job_id to track the crawling progress.
      tags:
        - - subpackage_website
      parameters:
        - name: domain
          in: path
          required: true
          schema:
            type: string
        - name: Authorization
          in: header
          description: >-
            Bearer authentication of the form `Bearer <token>`, where token is
            your auth token.
          required: true
          schema:
            type: string
      responses:
        '200':
          description: Successful Response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/IndexWebsiteResponse'
        '422':
          description: Validation Error
          content: {}
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/IndexWebsiteRequest'
components:
  schemas:
    IndexWebsiteRequest:
      type: object
      properties:
        base_url:
          type: string
          description: >-
            The base URL to start indexing from (e.g.,
            'https://docs.example.com')
        domain_filter:
          type:
            - string
            - 'null'
          description: >-
            Domain to filter crawling (e.g., 'docs.example.com'). Defaults to
            base_url domain.
        path_filter:
          type:
            - string
            - 'null'
          description: >-
            Path prefix to restrict crawling (e.g., '/docs'). Only URLs starting
            with this will be crawled.
        url_pattern:
          type:
            - string
            - 'null'
          description: >-
            Regex pattern to filter URLs (e.g.,
            `https://example\.com/(docs|api)/.*`).
        chunk_size:
          type:
            - integer
            - 'null'
          default: 1000
          description: Size of text chunks for splitting documents
        chunk_overlap:
          type:
            - integer
            - 'null'
          default: 200
          description: Overlap between consecutive chunks
        min_content_length:
          type:
            - integer
            - 'null'
          default: 100
          description: Minimum content length to index a page
        max_pages:
          type:
            - integer
            - 'null'
          description: Maximum number of pages to crawl. None means unlimited.
        delay:
          type:
            - number
            - 'null'
          format: double
          default: 1
          description: Delay in seconds between requests
        version:
          type:
            - string
            - 'null'
          description: Version to tag all indexed pages with
        product:
          type:
            - string
            - 'null'
          description: Product to tag all indexed pages with
        authed:
          type:
            - boolean
            - 'null'
          description: Whether indexed pages should be auth-gated
      required:
        - base_url
    IndexWebsiteResponse:
      type: object
      properties:
        job_id:
          type: string
          description: ID to track the indexing job status
        base_url:
          type: string
          description: The base URL being indexed
      required:
        - job_id
        - base_url

```