Skip to main content

Ingest Custom Document for Indexing

Overview​

Tabby provides an HTTP REST API that enables direct data ingestion into Tabby's index.

This API allows you to seamlessly integrate your own data sources, ensuring that Tabby's chat experiences are tailored to your specific requirements and context.

Ingested documents are organized hierarchically by source and then by id. Each document you provide must have a source string and a unique id string within that source.

This structure allows documents to be grouped logically under a common source. When you want to retrieve documents, you can specify a particular source, and Tabby will operate on the documents within that group.

Documents are created individually and can be deleted either one by one or in bulk by deleting all documents associated with a particular source.

Data Ingestion HTTP API​

The Data Ingestion API enables you to:

  • Add custom documents to Tabby's index with a configurable time-to-live (TTL)
  • Delete specific documents or all documents from a particular source

Note that all ingestion and deletion operations are processed asynchronously during scheduled index updates.

Authentication​

To use the Data Ingestion API, you must authenticate your requests. Tabby's HTTP API uses Registration API tokens for authentication, which could be retrieve as follow:

Registration Token

Include your API key in the Authorization request header as follows:

Authorization: Bearer YOUR_API_KEY

Add Document​

POST /v1beta/ingestion

Request Body:

{
  "source": "custom-documentation",
  "id": "doc-id",
  "title": "API Documentation",
  "body": "This is the content of the documentation...",
  "link": "https://example.com/docs/123",
  "ttl": "90d"
}
FieldTypeDescription
sourcestringRequired. Source identifier for the document. Used for grouping and deletion operations.
idstringRequired. Unique identifier for the document within the same source.
titlestringRequired. Title of the document.
bodystringRequired. Content of the document to be indexed.
linkstringOptional. URL or reference to the original document.
ttlstringOptional. Time-to-live duration (e.g., "90d" for 90 days). After this period, the document will be automatically removed during garbage collection.

Response:

  • 202 Accepted: The request has been accepted and will be processed later. 
    {
      "id": "doc123",
      "status": "pending",
      "message": "Your request has been accepted and will be processed later."
    }
  • 400 Bad Request: The request was invalid. 
    {
      "error": "Invalid request body",
      "message": "The request body is missing required fields or has invalid values."
    }
  • 500 Internal Error: The server encountered an error while processing the request. 
    {
      "error": "Internal Server Error",
      "message": "An unexpected error occurred while processing your request."
    }

Delete Document​

DELETE /v1beta/ingestion/{source}
DELETE /v1beta/ingestion/{source}/{id}

Deletes either all documents from a specific source (by source) or a specific document (by source and id).

Response:

  • 202 Accepted: The request has been accepted and will be processed later.
  • 401 Unauthorized: The request is missing or has an invalid API key.
  • 500 Internal Error: The server encountered an error while processing the request.

Monitoring Ingestion Status​

Since the ingestion process is asynchronous, you can monitor the status of your ingestion requests through the Tabby web UI, under the System tab in the Information section.

The ingestion status page offers a detailed summary of all your ingestion requests, including the total count, pending requests, and any failures.

Ingestion Status