Relevant source files
The following files were used as context for generating this wiki page:
- apps/api/src/knowledge-base/knowledge-base.service.ts
- apps/api/src/knowledge-base/knowledge-base.controller.ts
- apps/api/src/knowledge-base/utils/s3-operations.ts
- apps/api/src/knowledge-base/utils/constants.ts
- apps/api/src/knowledge-base/knowledge-base.module.ts
- apps/api/src/knowledge-base/dto/process-documents.dto.ts
- apps/api/src/knowledge-base/dto/upload-document.dto.ts
The Knowledge Base module provides a robust system for managing organizational documents and manual answers, integrating file storage, database persistence, and asynchronous processing capabilities. It allows users to upload, list, retrieve, and delete documents, as well as manage manual answers, with support for vector database integration for intelligent search and retrieval. The module leverages Amazon S3 for secure file storage and Trigger.dev for orchestrating background processing tasks.
This module is designed to handle various aspects of knowledge management, from initial document ingestion to their eventual processing and deletion, ensuring data consistency across storage, database, and vector store components.
Architecture Overview
The Knowledge Base module follows a standard NestJS architecture, with a Controller handling incoming API requests and a Service encapsulating the business logic. It interacts with external services such as S3 for file storage and Trigger.dev for asynchronous task orchestration, and persists metadata in a database.
Sources: apps/api/src/knowledge-base/knowledge-base.module.ts, apps/api/src/knowledge-base/knowledge-base.controller.ts, apps/api/src/knowledge-base/knowledge-base.service.ts
API Endpoints
The KnowledgeBaseController exposes a set of RESTful API endpoints for interacting with knowledge base documents and manual answers.
| Method | Endpoint | Description | DTO / Parameters The Knowledge Base module is a core component that manages the storage, processing, and retrieval of documents and manual answers for an organization. It integrates with Snbsp;S3 for file storage and Trigger.dev for asynchronous processing, providing a robust system for knowledge management. Sources: apps/api/src/knowledge-base/knowledge-base.module.ts, apps/api/src/knowledge-base/knowledge-base.service.ts
Architecture Overview
The Knowledge Base module follows a standard NestJS architecture, with a Controller handling incoming API requests and a Service encapsulating the business logic. It interacts with external services such as S3 for file storage and Trigger.dev for asynchronous task orchestration, and persists metadata in a database.
Sources: apps/api/src/knowledge-base/knowledge-base.module.ts, apps/api/src/knowledge-base/knowledge-base.controller.ts, apps/api/src/knowledge-base/knowledge-base.service.ts
API Endpoints
The KnowledgeBaseController exposes a set of RESTful API endpoints for interacting with knowledge base documents and manual answers.
| Method | Endpoint | Description | DTO / Parameters Sources: apps/api/src/knowledge-base/knowledge-base.controller.ts, apps/api/src/knowledge-base/knowledge-base.service.ts, apps/api/src/knowledge-base/utils/s3-operations.ts
Listing Documents
The listDocuments endpoint retrieves all knowledge base documents associated with a given organizationId. It fetches document metadata from the database, including id, name, description, s3Key, fileType, fileSize, processingStatus, createdAt, and updatedAt.
async listDocuments(organizationId: string) {
return db.knowledgeBaseDocument.findMany({
where: { organizationId },
select: {
id: true,
name: true,
description: true,
s3Key: true,
fileType: true,
fileSize: true,
processingStatus: true,
createdAt: true,
updatedAt: true,
},
orderBy: { createdAt: 'desc' },
});
}Sources: apps/api/src/knowledge-base/knowledge-base.service.ts:18-32, apps/api/src/knowledge-base/knowledge-base.controller.ts:40-59
Getting Signed URLs
The module provides functionality to generate time-limited, signed URLs for both downloading and viewing documents directly in a browser.
Download URL
The getDownloadUrl method generates a signed URL that forces the browser to download the file.
Sources: apps/api/src/knowledge-base/knowledge-base.service.ts:51-64, apps/api/src/knowledge-base/knowledge-base.controller.ts:76-92
View URL
The getViewUrl method generates a signed URL intended for inline viewing in the browser. It also indicates whether the file type is generally viewable in a browser based on predefined MIME types.
Sources: apps/api/src/knowledge-base/knowledge-base.service.ts:66-82, apps/api/src/knowledge-base/knowledge-base.controller.ts:94-115, apps/api/src/knowledge-base/utils/constants.ts:30-32
Deleting Documents
Deleting a document involves multiple steps to ensure consistency across S3, the database, and any associated vector store embeddings. This process is partially asynchronous to avoid blocking the API response.
Sources: apps/api/src/knowledge-base/knowledge-base.service.ts:84-122, apps/api/src/knowledge-base/knowledge-base.controller.ts:117-142, apps/api/src/knowledge-base/utils/s3-operations.ts:130-143
Processing Documents
The processDocuments endpoint initiates the asynchronous processing of one or more knowledge base documents. This typically involves extracting content and generating vector embeddings for search. The service intelligently dispatches either a single document processing task or an orchestrator task for multiple documents.
Sources: apps/api/src/knowledge-base/knowledge-base.service.ts:124-154, apps/api/src/knowledge-base/knowledge-base.controller.ts:144-162, apps/api/src/knowledge-base/dto/process-documents.dto.ts
Document processing tasks are handled asynchronously by Trigger.dev. The API immediately returns a runId and a publicAccessToken which can be used to monitor the status of the background job.
Manual Answer Management
The module also supports the management of "manual answers," which are likely pre-defined responses or knowledge entries.
Deleting a Single Manual Answer
The deleteManualAnswer method removes a specific manual answer from the system. It triggers an asynchronous task to delete associated vector embeddings before removing the record from the main database.
Sources: apps/api/src/knowledge-base/knowledge-base.service.ts:175-199, apps/api/src/knowledge-base/knowledge-base.controller.ts:182-196
Deleting All Manual Answers
The deleteAllManualAnswers method allows for the bulk deletion of all manual answers belonging to a specific organization. It first identifies all relevant manual answer IDs and then triggers an orchestrator task for batch vector deletion before removing all records from the database.
Sources: apps/api/src/knowledge-base/knowledge-base.service.ts:201-229, apps/api/src/knowledge-base/knowledge-base.controller.ts:198-207
S3 Operations
The s3-operations.ts utility file provides core functions for interacting with Amazon S3, specifically for the knowledge base bucket.
Key Functions
| Function Name | Description ```mermaid erDiagram knowledgeBaseDocument { string id string name string description string s3Key string fileType number fileSize string processingStatus string createdAt string updatedAt string organizationId
}securityQuestionnaireManualAnswer { string id string organizationId
}organization ||--o{ knowledgeBaseDocument : "manages" organization ||--o{ securityQuestionnaireManualAnswer : "manages"
Sources: [apps/api/src/knowledge-base/knowledge-base.service.ts:18-32](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/knowledge-base.service.ts#L18-L32), [apps/api/src/knowledge-base/knowledge-base.service.ts:175-181](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/knowledge-base.service.ts#L175-L181)
## Constants and Utilities
The `constants.ts` file defines various configuration parameters and helper functions used across the Knowledge Base module.
| Constant/Function Name | Description
Sources: [apps/api/src/knowledge-base/knowledge-base.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/knowledge-base.controller.ts), [apps/api/src/knowledge-base/knowledge-base.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/knowledge-base.service.ts), [apps/api/src/knowledge-base/utils/s3-operations.ts](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/utils/s3-operations.ts)
### Listing Documents
The `listDocuments` endpoint retrieves all knowledge base documents associated with a given `organizationId`. It fetches document metadata from the database, including `id`, `name`, `description`, `s3Key`, `fileType`, `fileSize`, `processingStatus`, `createdAt`, and `updatedAt`.
```typescript
async listDocuments(organizationId: string) {
return db.knowledgeBaseDocument.findMany({
where: { organizationId },
select: {
id: true,
name: true,
description: true,
s3Key: true,
fileType: true,
fileSize: true,
processingStatus: true,
createdAt: true,
updatedAt: true,
},
orderBy: { createdAt: 'desc' },
});
}
Sources: apps/api/src/knowledge-base/knowledge-base.service.ts:18-32, apps/api/src/knowledge-base/knowledge-base.controller.ts:40-59
Getting Signed URLs
The module provides functionality to generate time-limited, signed URLs for both downloading and viewing documents directly in a browser.
Download URL
The getDownloadUrl method generates a signed URL that forces the browser to download the file.
Sources: apps/api/src/knowledge-base/knowledge-base.service.ts:51-64, apps/api/src/knowledge-base/knowledge-base.controller.ts:76-92
View URL
The getViewUrl method generates a signed URL intended for inline viewing in the browser. It also indicates whether the file type is generally viewable in a browser based on predefined MIME types.
Sources: apps/api/src/knowledge-base/knowledge-base.service.ts:66-82, apps/api/src/knowledge-base/knowledge-base.controller.ts:94-115, apps/api/src/knowledge-base/utils/constants.ts:30-32
Deleting Documents
Deleting a document involves multiple
On This Page
Architecture OverviewAPI EndpointsArchitecture OverviewAPI EndpointsListing DocumentsGetting Signed URLsDownload URLView URLDeleting DocumentsProcessing DocumentsManual Answer ManagementDeleting a Single Manual AnswerDeleting All Manual AnswersS3 OperationsKey FunctionsGetting Signed URLsDownload URLView URLDeleting Documents