---
title: "Storage Flows"
description: "This page details the various storage flows within the application, primarily focusing on how files and data are managed using Amazon S3. It covers the core S3 client configuration, and specific im..."
last_updated: "2026-05-06T07:29:41.66289+00:00"
canonical_url: "https://www.doc0.dev/docs/49c89830-117b-4def-8edc-b5bcc50766e0/technical/section-4/storage-flows"
---
Relevant source files
The following files were used as context for generating this wiki page:
- [apps/api/src/attachments/attachments.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts)
- [apps/api/src/questionnaire/utils/questionnaire-storage.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts)
- [apps/api/src/knowledge-base/utils/s3-operations.ts](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/utils/s3-operations.ts)
- [apps/api/src/attachments/attachments.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.controller.ts)
- [apps/api/src/attachments/upload-attachment.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/upload-attachment.dto.ts)
- [apps/api/src/app/s3.ts](https://github.com/blade47/comp/blob/main/apps/api/src/app/s3.ts)
This page details the various storage flows within the application, primarily focusing on how files and data are managed using Amazon S3. It covers the core S3 client configuration, and specific implementations for handling attachments, questionnaire uploads, and knowledge base documents. The system leverages S3 for secure, scalable object storage and uses signed URLs for controlled access to private content.
The architecture ensures data segregation by organization and entity, robust security validations for file uploads, and persistence of metadata in a database alongside the S3 object storage.
## Core S3 Configuration and Client
The application initializes a global S3 client (`s3Client`) for all S3 interactions. This client is configured based on environment variables and is crucial for connecting to AWS S3 services.
### S3 Client Initialization
The `s3Client` is initialized once at application startup. It requires several environment variables to be set for proper functioning, including AWS credentials, region, and the default bucket name. If any critical configuration is missing, the client initialization fails, and a dummy client is created, indicating that S3 operations will not work.
The S3 client relies on the following environment variables. If these are not correctly set, S3 operations will fail:
- `APP_AWS_REGION`
- `APP_AWS_ACCESS_KEY_ID`
- `APP_AWS_SECRET_ACCESS_KEY`
- `APP_AWS_BUCKET_NAME` (aliased as `BUCKET_NAME`)
- `APP_AWS_ENDPOINT` (optional, for custom S3 endpoints)
Sources: [apps/api/src/app/s3.ts:10-44](https://github.com/blade47/comp/blob/main/apps/api/src/app/s3.ts#L10-L44)
### S3 Key Extraction and Validation
A utility function, `extractS3KeyFromUrl`, is provided to safely parse an S3 URL and extract its corresponding S3 key. This function includes security checks to prevent path traversal attacks and validates the S3 host.
```mermaid
sequenceDiagram
participant Caller
participant S3Utils as S3 Utilities (app/s3.ts)
Caller->>S3Utils: extractS3KeyFromUrl(url)
S3Utils->>S3Utils: Parse URL
alt URL is valid
S3Utils->>S3Utils: isValidS3Host(parsedUrl.host)
alt Host is valid S3 host
S3Utils->>S3Utils: Decode URI component from pathname
S3Utils->>S3Utils: Check for path traversal (../)
S3Utils-->>Caller: Return S3 Key
else Host is not valid S3 host
S3Utils-->>Caller: Throw "Invalid URL: Not a valid S3 endpoint"
end
else URL is malformed or not a URL
S3Utils->>S3Utils: Check for domain-like patterns
alt Domain-like pattern found
S3Utils-->>Caller: Throw "Invalid input: Domain-like pattern detected"
else No domain-like pattern
S3Utils->>S3Utils: Check for path traversal (../)
S3Utils-->>Caller: Return S3 Key (after '/' removal if present)
end
end
```
Sources: [apps/api/src/app/s3.ts:50-98](https://github.com/blade47/comp/blob/main/apps/api/src/app/s3.ts#L50-L98)
## Attachment Storage Flows
The `AttachmentsService` handles all operations related to file attachments, such as uploading, retrieving, and deleting files associated with various entities (e.g., tasks, policies).
### Attachment Upload Process
The `uploadAttachment` method is responsible for taking a base64 encoded file, validating it, uploading it to S3, and creating a corresponding record in the database.
**Key Steps:**
1. **Validation:** Checks for blocked file extensions and MIME types, and enforces a maximum file size (100MB).
2. **S3 Key Generation:** A unique S3 key is generated using the organization ID, entity type, entity ID, a timestamp, a random ID, and a sanitized file name. Special handling exists for `task_item` entity types to construct a more specific path.
* General pattern: `{organizationId}/attachments/{entityType}/{entityId}/{timestamp}-{fileId}-{sanitizedFileName}`
* Task Item pattern: `{organizationId}/attachments/task-item/{taskItemEntityType}/{taskItemEntityId}/{timestamp}-{fileId}-{sanitizedFileName}`
3. **S3 Upload:** The file buffer is uploaded to the configured S3 bucket using `PutObjectCommand`. Metadata such as `originalFileName`, `organizationId`, `entityId`, `entityType`, and `uploadedBy` are attached to the S3 object.
4. **Database Record:** A new record is created in the `db.attachment` table, storing the attachment's name, S3 URL (key), type (mapped from MIME type), entity ID, entity type, and organization ID.
5. **Signed URL Generation:** A temporary, signed download URL is generated for immediate access, expiring after 15 minutes (`SIGNED_URL_EXPIRY`).
Sources: [apps/api/src/attachments/attachments.service.ts:32-132](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts#L32-L132)
#### UploadAttachmentDto
This DTO defines the structure for attachment upload requests, including validation rules.
| Field | Type | Description
This document is based on the provided source code files. Any information not explicitly present in these files, such as detailed database schemas or external configurations not referenced, is outside the scope of this wiki page.
## Introduction
This document outlines the storage flows implemented within the API, focusing on how various types of files and data are managed using Amazon S3. The primary goal is to provide a robust, secure, and scalable solution for handling user-uploaded content, system-generated documents, and knowledge base assets. The system utilizes S3 for object storage, coupled with database records for metadata management, and employs signed URLs for controlled access to private resources.
The storage architecture is designed to support different application domains, including attachments for tasks and policies, questionnaire uploads, and knowledge base documents, each with its specific S3 key structure and access patterns. Security measures, such as file type validation and path traversal prevention, are integrated throughout these flows.
## S3 Client and Configuration
The application's interaction with Amazon S3 is centralized through a singleton `S3Client` instance, initialized in `apps/api/src/app/s3.ts`. This client is configured using environment variables, ensuring flexibility across different deployment environments.
### S3 Client Initialization Details
The `S3Client` is instantiated with the following parameters:
* `endpoint`: Optional, used for custom S3-compatible services.
* `region`: The AWS region where the S3 bucket resides, e.g., `us-east-1`.
* `credentials`: AWS access key ID and secret access key for authentication.
* `forcePathStyle`: Set to `true` if `APP_AWS_ENDPOINT` is provided, which is common for local S3 emulators.
A `Logger` is used to report initialization status and errors. If essential environment variables (`APP_AWS_ACCESS_KEY_ID`, `APP_AWS_SECRET_ACCESS_KEY`, `BUCKET_NAME`, `APP_AWS_REGION`) are missing, the S3 client will not be properly initialized, leading to a fallback to a dummy client and logging an error.
**Key Configuration Variables:**
* `BUCKET_NAME`: The default S3 bucket for general attachments.
* `APP_AWS_QUESTIONNAIRE_UPLOAD_BUCKET`: Specific bucket for questionnaire uploads.
* `APP_AWS_KNOWLEDGE_BASE_BUCKET`: Specific bucket for knowledge base documents.
* `APP_AWS_ORG_ASSETS_BUCKET`: Bucket for organization-specific assets.
Sources: [apps/api/src/app/s3.ts:10-44](https://github.com/blade47/comp/blob/main/apps/api/src/app/s3.ts#L10-L44)
### S3 Key Extraction and Validation
The `extractS3KeyFromUrl` function provides a secure way to obtain an S3 object key from a given URL. It performs several checks:
* **URL Parsing:** Attempts to parse the input as a URL.
* **S3 Host Validation:** Uses `isValidS3Host` to ensure the URL's host belongs to a legitimate AWS S3 domain.
* **Path Traversal Prevention:** Checks for `../` or `..\` patterns in the extracted key to prevent directory traversal vulnerabilities.
* **Empty Key Check:** Ensures the extracted key is not empty.
This function is critical for securely handling S3 URLs provided by external sources or users.
Sources: [apps/api/src/app/s3.ts:50-98](https://github.com/blade47/comp/blob/main/apps/api/src/app/s3.ts#L50-L98)
## Attachment Management
The `AttachmentsService` (`apps/api/src/attachments/attachments.service.ts`) is responsible for managing file attachments across various entities within the application. This includes uploading, retrieving, and deleting attachments, as well as generating secure download URLs.
### Attachment Upload Flow
The `uploadAttachment` method handles the end-to-end process of storing a new attachment.
Sources:
- [apps/api/src/attachments/attachments.service.ts:32-132](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts#L32-L132)
- [apps/api/src/attachments/attachments.controller.ts:20-20](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.controller.ts#L20-L20)
#### File Validation
The service implements strict validation for uploaded files:
* **Blocked Extensions:** A list of executable or potentially dangerous file extensions (e.g., `.exe`, `.bat`, `.js`, `.sh`) is blocked.
* **Blocked MIME Types:** A list of dangerous MIME types (e.g., `application/x-msdownload`, `application/javascript`) is blocked.
* **File Size Limit:** Files are limited to `100 MB` (`MAX_FILE_SIZE_BYTES`).
Sources: [apps/api/src/attachments/attachments.service.ts:39-77](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts#L39-L77)
#### S3 Key Structure
The S3 key for attachments follows a structured path to ensure organization and easy retrieval:
* General: `{organizationId}/attachments/{entityType}/{entityId}/{timestamp}-{fileId}-{sanitizedFileName}`
* For `task_item` entity type: `{organizationId}/attachments/task-item/{taskItemEntityType}/{taskItemEntityId}/{timestamp}-{fileId}-{sanitizedFileName}`. The `taskItemEntityType` and `taskItemEntityId` are extracted from the `description` field of the `UploadAttachmentDto`.
Sources: [apps/api/src/attachments/attachments.service.ts:80-92](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts#L80-L92)
#### UploadAttachmentDto
The data transfer object for attachment uploads includes:
| Field | Type | Description
### Attachment Download Flow
The `AttachmentsController` exposes an endpoint to generate a signed URL for an attachment.
```mermaid
sequenceDiagram
participant Client
participant AttachmentsController
participant AttachmentsService
participant S3Client as AWS S3
Client->>AttachmentsController: GET /attachments/:attachmentId/download
AttachmentsController->>AttachmentsService: getAttachmentDownloadUrl(orgId, attachmentId)
AttachmentsService->>db: findFirst({ id: attachmentId, organizationId })
db-->>AttachmentsService: Attachment Record
alt Attachment not found
AttachmentsService-->>AttachmentsController: Throw BadRequestException
AttachmentsController-->>Client: 400 Bad Request
else Attachment found
AttachmentsService->>AttachmentsService: generateSignedUrl(attachment.url)
AttachmentsService->>S3Client: getSignedUrl(GetObjectCommand)
S3Client-->>AttachmentsService: Signed URL
AttachmentsService-->>AttachmentsController: { downloadUrl, expiresIn }
AttachmentsController-->>Client: 200 OK { downloadUrl, expiresIn }
end
```
Sources:
- [apps/api/src/attachments/attachments.controller.ts:39-60](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.controller.ts#L39-L60)
- [apps/api/src/attachments/attachments.service.ts:149-178](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts#L149-L178)
#### Signed URL Generation
The `generateSignedUrl` private method uses `@aws-sdk/s3-request-presigner` to create temporary URLs for `GetObjectCommand` requests. These URLs allow direct download from S3 without requiring AWS credentials, and they expire after a predefined duration (`SIGNED_URL_EXPIRY`, 15 minutes). The `getPresignedDownloadUrlWithFilename` method allows specifying a custom filename for the download.
Sources: [apps/api/src/attachments/attachments.service.ts:249-256](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts#L249-L256), [apps/api/src/attachments/attachments.service.ts:265-277](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts#L265-L277)
### Other Attachment Operations
* **`getAttachments`**: Retrieves all attachment metadata for a given entity, including generated signed URLs for each.
* **`getAttachmentMetadata`**: Retrieves attachment metadata without generating signed URLs, useful for displaying lists of attachments.
* **`deleteAttachment`**: Deletes an attachment from both S3 and the database.
* **`copyPolicyVersionPdf`**: Copies an S3 object (policy PDF) to a new key, used for versioning.
* **`deletePolicyVersionPdf`**: Deletes a specific policy version PDF from S3.
* **`uploadToS3`**: A generic S3 upload method used by other services (e.g., `policies.service.ts`, `evidence-forms.service.ts`).
* **`getObjectBuffer`**: Retrieves an S3 object's content as a Buffer.
Sources: [apps/api/src/attachments/attachments.service.ts:135-246](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts#L135-L246), [apps/api/src/attachments/attachments.service.ts:258-261](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts#L258-L261), [apps/api/src/attachments/attachments.service.ts:279-293](https://github.com/blade47/comp/blob/main/apps/api/src/attachments/attachments.service.ts#L279-L293)
## Questionnaire Storage Flows
The `questionnaire-storage.ts` utility file provides functions for handling questionnaire file uploads to S3 and persisting questionnaire results and answers to the database.
### Questionnaire File Upload
The `uploadQuestionnaireFile` function handles the storage of questionnaire files.
Sources: [apps/api/src/questionnaire/utils/questionnaire-storage.ts:90-129](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts#L90-L129)
**S3 Key Structure:**
The S3 key for questionnaire uploads follows the pattern: `{organizationId}/questionnaire-uploads/{timestamp}-{fileId}-{sanitizedFileName}`.
Sources: [apps/api/src/questionnaire/utils/questionnaire-storage.ts:114-114](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts#L114-L114)
### Questionnaire Result Persistence
The `persistQuestionnaireResult` function saves the parsed questionnaire data and answers to the database.
Sources: [apps/api/src/questionnaire/utils/questionnaire-storage.ts:48-87](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts#L48-L87)
### Answer Management
* **`saveGeneratedAnswer`**: Updates an existing question's answer or creates a new one in the `questionnaireQuestionAnswer` table.
* **`updateAnsweredCount`**: Recalculates and updates the `answeredQuestions` count for a given questionnaire in the `questionnaire` table.
Sources: [apps/api/src/questionnaire/utils/questionnaire-storage.ts:132-172](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts#L132-L172), [apps/api/src/questionnaire/utils/questionnaire-storage.ts:34-45](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts#L34-L45)
## Knowledge Base Storage Flows
The `s3-operations.ts` file within the knowledge base module provides dedicated functions for managing knowledge base documents in S3.
### Knowledge Base Document Upload
The `uploadToS3` function handles the upload of knowledge base documents.
Sources: [apps/api/src/knowledge-base/utils/s3-operations.ts:34-66](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/utils/s3-operations.ts#L34-L66)
**Key Configuration:**
* `APP_AWS_KNOWLEDGE_BASE_BUCKET`: The dedicated S3 bucket for knowledge base documents.
* `MAX_FILE_SIZE_BYTES`: Maximum allowed file size for knowledge base documents.
Sources: [apps/api/src/knowledge-base/utils/s3-operations.ts:13-13](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/utils/s3-operations.ts#L13-L13), [apps/api/src/knowledge-base/utils/s3-operations.ts:10-10](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/utils/s3-operations.ts#L10-L10)
### Knowledge Base Document Access and Deletion
* **`generateDownloadUrl`**: Creates a signed URL for downloading a knowledge base document. The `ResponseContentDisposition` is set to `attachment` to prompt a download.
* **`generateViewUrl`**: Creates a signed URL for viewing a knowledge base document directly in the browser. The `ResponseContentDisposition` is set to `inline`.
* **`deleteFromS3`**: Deletes a document from the `APP_AWS_KNOWLEDGE_BASE_BUCKET`. It returns `true` on success and `false` on error, without throwing exceptions.
* **`validateS3Config`**: Ensures that the S3 client is initialized and the `APP_AWS_KNOWLEDGE_BASE_BUCKET` environment variable is set.
Sources: [apps/api/src/knowledge-base/utils/s3-operations.ts:69-122](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/utils/s3-operations.ts#L69-L122), [apps/api/src/knowledge-base/utils/s3-operations.ts:24-32](https://github.com/blade47/comp/blob/main/apps/api/src/knowledge-base/utils/s3-operations.ts#L24-L32)
## Sitemap
See the full [sitemap](https://www.doc0.dev/docs/49c89830-117b-4def-8edc-b5bcc50766e0/llms.txt) for all pages in this wiki.