System Architecture
Core Features
Data Management
Frontend Components
Extensibility
<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:
- [apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts)
- [apps/api/src/questionnaire/questionnaire.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.controller.ts)
- [apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts)
- [apps/api/src/questionnaire/utils/questionnaire-storage.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts)
- [apps/api/src/questionnaire/utils/question-parser.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/question-parser.ts)
- [apps/api/src/questionnaire/questionnaire.module.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.module.ts)
- [apps/api/src/questionnaire/dto/auto-answer.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/auto-answer.dto.ts)
- [apps/api/src/questionnaire/dto/upload-and-parse.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/upload-and-parse.dto.ts)
- [apps/api/src/questionnaire/utils/deduplicate-sources.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/deduplicate-sources.ts)
- [apps/api/src/questionnaire/utils/export-generator.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/export-generator.ts)
</details>
The Questionnaire module provides a comprehensive solution for processing, answering, and managing security questionnaires. It enables users to upload various file formats (e.g., PDF, Excel, CSV, images), automatically extract questions and answers using AI, generate answers based on an organization's knowledge base, and export the results in multiple formats.
This module is designed to streamline the often time-consuming process of responding to security questionnaires, leveraging advanced AI models and a robust data storage and retrieval system. It supports both internal users and external trust portal access for automated questionnaire processing.
## Architecture Overview
The Questionnaire module is built around a service-controller pattern, utilizing several utility modules for specific tasks such as content extraction, question parsing, storage, and export generation.
<Callout title="Key Components" variant="info">
The core components include:
- `QuestionnaireController`: Handles incoming API requests, authentication, and response formatting.
- `QuestionnaireService`: Orchestrates the business logic, interacting with AI models, database, and S3 storage.
- `ContentExtractor`: Responsible for extracting raw content from various file types and performing initial AI-powered question/answer parsing.
- `QuestionnaireStorage`: Manages file uploads to S3 and persistence of questionnaire data (questions, answers, metadata) to the database.
- `ExportGenerator`: Creates export files in different formats (XLSX, CSV, PDF).
- `DeduplicateSources`: Utility for cleaning up and presenting RAG sources.
</Callout>
Sources:
[apps/api/src/questionnaire/questionnaire.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.controller.ts#L32-L36)
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L34-L46)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L43-L46)
[apps/api/src/questionnaire/utils/questionnaire-storage.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts#L22-L25)
[apps/api/src/questionnaire/utils/export-generator.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/export-generator.ts#L17-L20)
## Questionnaire Parsing and Upload
The module supports uploading questionnaire files in various formats and extracting their content, then parsing questions and answers using AI.
### File Upload and Content Extraction
Users can upload files via dedicated API endpoints. The `QuestionnaireService` delegates the initial file handling to `QuestionnaireStorage` for S3 upload and then to `ContentExtractor` for processing.
The `ContentExtractor` module is central to this process. It identifies the file type and employs different strategies:
* **Excel (XLSX, XLS):** Uses `AdmZip` and `XLSX` libraries for raw content extraction, including custom logic to handle rich text and shared strings often missed by standard parsers.
* **CSV:** Simple text extraction.
* **Text:** Direct text extraction.
* **PDF and Images (PNG, JPG):** Leverages OpenAI's Vision API (`gpt-4o`) to extract text and structure from visual documents.
* **Word Documents (DOCX):** Currently not directly supported for parsing and advises conversion to PDF or image.
After raw content extraction, `extractQuestionsWithAI` orchestrates the AI-powered parsing:
* **Groq (`gpt-oss-120b`):** Primary and fastest model for parsing questions and answers from textual content, especially for Excel and CSV. It uses a chunking strategy for large files.
* **Claude (`claude-3-5-sonnet-latest`):** Fallback for Groq, offering excellent quality with a larger context window.
* **OpenAI (`gpt-4o-mini`):** Further fallback for general text parsing.
* **OpenAI Vision (`gpt-4o`):** Used specifically for PDF and image files.
The AI models are prompted to extract both traditional questions and form-style fields (e.g., "Vendor Name", "Contact Email") along with their corresponding answers.
Sources:
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L56-L63)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L43-L121)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L125-L132)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L140-L144)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L160-L200)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L204-L215)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L220-L245)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L250-L275)
### API Endpoints for Parsing and Upload
| Method | Endpoint | Description
<Callout title="Important Note" variant="info">
The `question-parser.ts` file contains utilities for parsing questions and answers from content. While it defines interfaces and helper functions, the primary AI-powered parsing logic for the main questionnaire service flow is handled by `content-extractor.ts` through its `extractQuestionsWithAI` function, which internally utilizes various AI models.
</Callout>
Sources:
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L56-L63)
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L104-L110)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L125-L132)
| Method | Endpoint | Description
classDiagram
class QuestionnaireService {
+parseQuestionnaire(dto: ParseQuestionnaireDto): ParsedQuestionnaireResult
+autoAnswerAndExport(dto: ExportQuestionnaireDto): QuestionnaireExportResult
+uploadAndParse(dto: UploadAndParseDto): { questionnaireId: string; totalQuestions: number }
+answerSingleQuestion(dto: AnswerSingleQuestionDto): AnswerQuestionResult
+saveAnswer(dto: SaveAnswerDto): { success: boolean; error?: string }
+exportById(dto: ExportByIdDto): ExportResult
+deleteAnswer(dto: DeleteAnswerDto): { success: boolean; error?: string }
+saveGeneratedAnswerPublic(params: { questionnaireId: string; questionIndex: number; answer: string; sources?: AnswerQuestionResult['sources'] }): void
-generateAnswersForQuestions(questionsAndAnswers: QuestionnaireAnswer[], organizationId: string): QuestionnaireAnswer[]
}
class QuestionnaireController {
+parseQuestionnaire(dto: ParseQuestionnaireDto): ParsedQuestionnaireResult
+answerSingleQuestion(dto: AnswerSingleQuestionDto): any
+saveAnswer(dto: SaveAnswerDto): { success: boolean; error?: string }
+deleteAnswer(dto: DeleteAnswerDto): { success: boolean; error?: string }
+exportById(dto: ExportByIdDto, res: Response): Promise<void>
+uploadAndParse(dto: UploadAndParseDto): { questionnaireId: string; totalQuestions: number }
+uploadAndParseUpload(file: Express.Multer.File, body: { organizationId: string; source?: 'internal' | 'external' }): any
+parseQuestionnaireUpload(file: Express.Multer.File, body: { organizationId: string; format?: 'pdf' | 'csv' | 'xlsx'; source?: 'internal' | 'external' }, res: Response): Promise<void>
+parseQuestionnaireUploadByToken(file: Express.Multer.File, token: string, body: { format?: 'pdf' | 'csv' | 'xlsx' }, res: Response): Promise<void>
+autoAnswerAndExport(dto: ExportQuestionnaireDto, res: Response): Promise<void>
+autoAnswerAndExportUpload(file: Express.Multer.File, body: { organizationId: string; format?: 'pdf' | 'csv' | 'xlsx' }, res: Response): Promise<void>
+autoAnswer(dto: AutoAnswerDto, res: Response): Promise<void>
}
class ParseQuestionnaireDto {
+vendorName?: string
+fileName?: string
+fileType: string
+fileData: string
}
class ExportQuestionnaireDto {
+organizationId: string
+fileData: string
+fileType: string
+fileName?: string
+vendorName?: string
+format: 'pdf' | 'csv' | 'xlsx'
+source?: 'internal' | 'external'
+exportInAllExtensions?: boolean
}
class AnswerSingleQuestionDto {
+questionnaireId: string
+organizationId: string
+question: string
+questionIndex: number
+totalQuestions: number
}
class AutoAnswerDto {
+organizationId: string
+questionnaireId?: string
+questionsAndAnswers: AutoAnswerQuestionDto[]
}
class AutoAnswerQuestionDto {
+question: string
+answer?: string | null
+_originalIndex?: number
}
class SaveAnswerDto {
+questionnaireId: string
+organizationId: string
+questionAnswerId?: string
+questionIndex?: number
+answer?: string | null
+status: 'manual' | 'generated'
+sources?: any
}
class DeleteAnswerDto {
+questionnaireId: string
+organizationId: string
+questionAnswerId: string
}
class UploadAndParseDto {
+organizationId: string
+fileName: string
+fileType: string
+fileData: string
+source?: 'internal' | 'external'
}
class ExportByIdDto {
+questionnaireId: string
+organizationId: string
+format: 'pdf' | 'csv' | 'xlsx'
}
class QuestionnaireAnswer {
+question: string
+answer: string | null
+sources?: any
}
class ParsedQuestionnaireResult {
+vendorName?: string
+fileName?: string
+totalQuestions: number
+questionsAndAnswers: QuestionnaireAnswer[]
}
class QuestionnaireExportResult {
+fileBuffer: Buffer
+mimeType: string
+filename: string
+questionsAndAnswers: QuestionnaireAnswer[]
}
QuestionnaireController --|> QuestionnaireService : uses
QuestionnaireService ..> ParseQuestionnaireDto
QuestionnaireService ..> ExportQuestionnaireDto
QuestionnaireService ..> AnswerSingleQuestionDto
QuestionnaireService ..> SaveAnswerDto
QuestionnaireService ..> DeleteAnswerDto
QuestionnaireService ..> UploadAndParseDto
QuestionnaireService ..> ExportByIdDto
QuestionnaireService ..> QuestionnaireAnswer
QuestionnaireService ..> ParsedQuestionnaireResult
QuestionnaireService ..> QuestionnaireExportResult
AutoAnswerDto o-- AutoAnswerQuestionDto : contains
\`\`\`
Sources:
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L34-L46)
[apps/api/src/questionnaire/questionnaire.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.controller.ts#L32-L36)
[apps/api/src/questionnaire/dto/parse-questionnaire.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/parse-questionnaire.dto.ts)
[apps/api/src/questionnaire/dto/export-questionnaire.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/export-questionnaire.dto.ts)
[apps/api/src/questionnaire/dto/answer-single-question.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/answer-single-question.dto.ts)
[apps/api/src/questionnaire/dto/auto-answer.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/auto-answer.dto.ts)
[apps/api/src/questionnaire/dto/save-answer.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/save-answer.dto.ts)
[apps/api/src/questionnaire/dto/delete-answer.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/delete-answer.dto.ts)
[apps/api/src/questionnaire/dto/upload-and-parse.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/upload-and-parse.dto.ts)
[apps/api/src/questionnaire/dto/export-by-id.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/export-by-id.dto.ts)
## Automatic Answering
The module provides robust capabilities for automatically generating answers to questionnaire questions using a Retrieval Augmented Generation (RAG) approach.
### Answer Generation Process
The `QuestionnaireService` orchestrates the automatic answering process:
1. **Sync Organization Embeddings:** Before generating answers, the system ensures that the organization's knowledge base (policies, manual answers, etc.) is synchronized with the vector store. This step is crucial for accurate retrieval.
2. **Batch Search:** For efficiency, questions are batched, and a single call to `findSimilarContentBatch` is made to retrieve relevant context from the vector store for all questions simultaneously.
3. **Answer Generation:** For each question, `generateAnswerFromContent` is called, which uses the retrieved context and an AI model (e.g., `generateAnswerWithRAGBatch` in the service) to formulate an answer.
4. **Save Answer:** Generated answers, along with their sources, are saved to the database.
5. **Update Answered Count:** The total count of answered questions for the questionnaire is updated.
### Streaming Auto-Answer (SSE)
The `autoAnswer` endpoint in the `QuestionnaireController` provides a Server-Sent Events (SSE) stream to give real-time feedback on the answer generation progress.
<Steps>
<Step>
### Establish SSE Connection
The controller sets up SSE headers and creates a safe sender function to stream events back to the client.
</Step>
<Step>
### Sync Embeddings
The `QuestionnaireService` first attempts to synchronize the organization's embeddings to ensure the vector store is up-to-date. Warnings are logged if this fails, but the process continues.
</Step>
<Step>
### Filter Unanswered Questions
The incoming list of questions is filtered to identify those that still require an answer.
</Step>
<Step>
### Batch Context Search
A progress event (`type: 'progress'`, `phase: 'searching'`) is sent. The system then performs a batch search (`findSimilarContentBatch`) against the vector store to retrieve relevant content for all unanswered questions. This is a critical optimization to reduce latency.
</Step>
<Step>
### Parallel Answer Generation
Another progress event (`type: 'progress'`, `phase: 'generating'`) is sent. Answers are then generated in parallel for each question using the pre-fetched content (`generateAnswerFromContent`).
</Step>
<Step>
### Stream Individual Answers
As each answer is generated, an `answer` event is streamed back to the client, including the question, generated answer, sources, and success status.
</Step>
<Step>
### Save Generated Answers
Each generated answer is persisted to the database via `saveGeneratedAnswerPublic` in the `QuestionnaireService`.
</Step>
<Step>
### Complete Stream
Once all questions are processed, a `complete` event is sent, summarizing the total and answered questions. The SSE connection is then closed.
</Step>
</Steps>
Sources:
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L225-L269)
[apps/api/src/questionnaire/questionnaire.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.controller.ts#L368-L476)
[apps/api/src/vector-store/lib/index.ts](https://github.com/blade47/comp/blob/main/apps/api/src/vector-store/lib/index.ts)
[apps/api/src/trigger/questionnaire/answer-question-helpers.ts](https://github.com/blade47/comp/blob/main/apps/api/src/trigger/questionnaire/answer-question-helpers.ts)
\`\`\`mermaid
sequenceDiagram
actor Client
participant Controller as QuestionnaireController
participant Service as QuestionnaireService
participant VectorStore as Vector Store
participant AI as AI Models
participant DB as Database
participant S3 as S3 Storage
Client->>Controller: POST /auto-answer (AutoAnswerDto)
Controller->>Controller: setupSSEHeaders()
Controller->>Service: syncOrganizationEmbeddings(orgId)
Service->>VectorStore: syncOrganizationEmbeddings(orgId)
VectorStore-->>Service: Sync Status
Service-->>Controller:
Controller->>Client: SSE: progress (phase: searching)
Controller->>VectorStore: findSimilarContentBatch(questions, orgId)
VectorStore-->>Controller: allSimilarContent[]
Controller->>Client: SSE: progress (phase: generating)
loop For each question
Controller->>AI: generateAnswerFromContent(question, similarContent)
AI-->>Controller: AnswerResult {answer, sources}
Controller->>Service: saveGeneratedAnswerPublic(questionnaireId, questionIndex, answer, sources)
Service->>DB: Update questionnaireQuestionAnswer
DB-->>Service:
Service->>QuestionnaireStorage: updateAnsweredCount(questionnaireId)
QuestionnaireStorage->>DB: Update questionnaire.answeredQuestions
DB-->>QuestionnaireStorage:
QuestionnaireStorage-->>Service:
Service-->>Controller:
Controller->>Client: SSE: answer {questionIndex, answer, sources}
end
Controller->>Client: SSE: complete {total, answered, answers}
Controller->>Client: Close SSE connection
\`\`\`
Sources:
[apps/api/src/questionnaire/questionnaire.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.controller.ts#L368-L476)
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L225-L269)
[apps/api/src/vector-store/lib/index.ts](https://github.com/blade47/comp/blob/main/apps/api/src/vector-store/lib/index.ts)
[apps/api/src/trigger/questionnaire/answer-question-helpers.ts](https://github.com/blade47/comp/blob/main/apps/api/src/trigger/questionnaire/answer-question-helpers.ts)
[apps/api/src/questionnaire/utils/questionnaire-storage.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts#L22-L36)
### DTOs for Auto-Answering
| DTO Class | Description
\`\`\`