Skip to main content
doc0blade47/compdocumentation home
Sign in

Questionnaires

Last updated Oct 11, 2026
<details>
<summary>Relevant source files</summary>

The following files were used as context for generating this wiki page:

- [apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts)
- [apps/api/src/questionnaire/questionnaire.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.controller.ts)
- [apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts)
- [apps/api/src/questionnaire/utils/questionnaire-storage.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts)
- [apps/api/src/questionnaire/utils/question-parser.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/question-parser.ts)
- [apps/api/src/questionnaire/questionnaire.module.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.module.ts)
- [apps/api/src/questionnaire/dto/auto-answer.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/auto-answer.dto.ts)
- [apps/api/src/questionnaire/dto/upload-and-parse.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/upload-and-parse.dto.ts)
- [apps/api/src/questionnaire/utils/deduplicate-sources.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/deduplicate-sources.ts)
- [apps/api/src/questionnaire/utils/export-generator.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/export-generator.ts)
</details>

The Questionnaire module provides a comprehensive solution for processing, answering, and managing security questionnaires. It enables users to upload various file formats (e.g., PDF, Excel, CSV, images), automatically extract questions and answers using AI, generate answers based on an organization's knowledge base, and export the results in multiple formats.

This module is designed to streamline the often time-consuming process of responding to security questionnaires, leveraging advanced AI models and a robust data storage and retrieval system. It supports both internal users and external trust portal access for automated questionnaire processing.

## Architecture Overview

The Questionnaire module is built around a service-controller pattern, utilizing several utility modules for specific tasks such as content extraction, question parsing, storage, and export generation.

<Callout title="Key Components" variant="info">
The core components include:
- `QuestionnaireController`: Handles incoming API requests, authentication, and response formatting.
- `QuestionnaireService`: Orchestrates the business logic, interacting with AI models, database, and S3 storage.
- `ContentExtractor`: Responsible for extracting raw content from various file types and performing initial AI-powered question/answer parsing.
- `QuestionnaireStorage`: Manages file uploads to S3 and persistence of questionnaire data (questions, answers, metadata) to the database.
- `ExportGenerator`: Creates export files in different formats (XLSX, CSV, PDF).
- `DeduplicateSources`: Utility for cleaning up and presenting RAG sources.
</Callout>


Sources:
[apps/api/src/questionnaire/questionnaire.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.controller.ts#L32-L36)
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L34-L46)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L43-L46)
[apps/api/src/questionnaire/utils/questionnaire-storage.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts#L22-L25)
[apps/api/src/questionnaire/utils/export-generator.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/export-generator.ts#L17-L20)

## Questionnaire Parsing and Upload

The module supports uploading questionnaire files in various formats and extracting their content, then parsing questions and answers using AI.

### File Upload and Content Extraction

Users can upload files via dedicated API endpoints. The `QuestionnaireService` delegates the initial file handling to `QuestionnaireStorage` for S3 upload and then to `ContentExtractor` for processing.

The `ContentExtractor` module is central to this process. It identifies the file type and employs different strategies:
*   **Excel (XLSX, XLS):** Uses `AdmZip` and `XLSX` libraries for raw content extraction, including custom logic to handle rich text and shared strings often missed by standard parsers.
*   **CSV:** Simple text extraction.
*   **Text:** Direct text extraction.
*   **PDF and Images (PNG, JPG):** Leverages OpenAI's Vision API (`gpt-4o`) to extract text and structure from visual documents.
*   **Word Documents (DOCX):** Currently not directly supported for parsing and advises conversion to PDF or image.

After raw content extraction, `extractQuestionsWithAI` orchestrates the AI-powered parsing:
*   **Groq (`gpt-oss-120b`):** Primary and fastest model for parsing questions and answers from textual content, especially for Excel and CSV. It uses a chunking strategy for large files.
*   **Claude (`claude-3-5-sonnet-latest`):** Fallback for Groq, offering excellent quality with a larger context window.
*   **OpenAI (`gpt-4o-mini`):** Further fallback for general text parsing.
*   **OpenAI Vision (`gpt-4o`):** Used specifically for PDF and image files.

The AI models are prompted to extract both traditional questions and form-style fields (e.g., "Vendor Name", "Contact Email") along with their corresponding answers.


Sources:
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L56-L63)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L43-L121)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L125-L132)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L140-L144)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L160-L200)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L204-L215)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L220-L245)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L250-L275)

### API Endpoints for Parsing and Upload

| Method | Endpoint                       | Description 
        <Callout title="Important Note" variant="info">
            The `question-parser.ts` file contains utilities for parsing questions and answers from content. While it defines interfaces and helper functions, the primary AI-powered parsing logic for the main questionnaire service flow is handled by `content-extractor.ts` through its `extractQuestionsWithAI` function, which internally utilizes various AI models.
        </Callout>
Sources:
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L56-L63)
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L104-L110)
[apps/api/src/questionnaire/utils/content-extractor.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/content-extractor.ts#L125-L132)

| Method | Endpoint                               | Description 
classDiagram
    class QuestionnaireService {
        +parseQuestionnaire(dto: ParseQuestionnaireDto): ParsedQuestionnaireResult
        +autoAnswerAndExport(dto: ExportQuestionnaireDto): QuestionnaireExportResult
        +uploadAndParse(dto: UploadAndParseDto): { questionnaireId: string; totalQuestions: number }
        +answerSingleQuestion(dto: AnswerSingleQuestionDto): AnswerQuestionResult
        +saveAnswer(dto: SaveAnswerDto): { success: boolean; error?: string }
        +exportById(dto: ExportByIdDto): ExportResult
        +deleteAnswer(dto: DeleteAnswerDto): { success: boolean; error?: string }
        +saveGeneratedAnswerPublic(params: { questionnaireId: string; questionIndex: number; answer: string; sources?: AnswerQuestionResult['sources'] }): void
        -generateAnswersForQuestions(questionsAndAnswers: QuestionnaireAnswer[], organizationId: string): QuestionnaireAnswer[]
    }

    class QuestionnaireController {
        +parseQuestionnaire(dto: ParseQuestionnaireDto): ParsedQuestionnaireResult
        +answerSingleQuestion(dto: AnswerSingleQuestionDto): any
        +saveAnswer(dto: SaveAnswerDto): { success: boolean; error?: string }
        +deleteAnswer(dto: DeleteAnswerDto): { success: boolean; error?: string }
        +exportById(dto: ExportByIdDto, res: Response): Promise<void>
        +uploadAndParse(dto: UploadAndParseDto): { questionnaireId: string; totalQuestions: number }
        +uploadAndParseUpload(file: Express.Multer.File, body: { organizationId: string; source?: 'internal' | 'external' }): any
        +parseQuestionnaireUpload(file: Express.Multer.File, body: { organizationId: string; format?: 'pdf' | 'csv' | 'xlsx'; source?: 'internal' | 'external' }, res: Response): Promise<void>
        +parseQuestionnaireUploadByToken(file: Express.Multer.File, token: string, body: { format?: 'pdf' | 'csv' | 'xlsx' }, res: Response): Promise<void>
        +autoAnswerAndExport(dto: ExportQuestionnaireDto, res: Response): Promise<void>
        +autoAnswerAndExportUpload(file: Express.Multer.File, body: { organizationId: string; format?: 'pdf' | 'csv' | 'xlsx' }, res: Response): Promise<void>
        +autoAnswer(dto: AutoAnswerDto, res: Response): Promise<void>
    }

    class ParseQuestionnaireDto {
        +vendorName?: string
        +fileName?: string
        +fileType: string
        +fileData: string
    }

    class ExportQuestionnaireDto {
        +organizationId: string
        +fileData: string
        +fileType: string
        +fileName?: string
        +vendorName?: string
        +format: 'pdf' | 'csv' | 'xlsx'
        +source?: 'internal' | 'external'
        +exportInAllExtensions?: boolean
    }

    class AnswerSingleQuestionDto {
        +questionnaireId: string
        +organizationId: string
        +question: string
        +questionIndex: number
        +totalQuestions: number
    }

    class AutoAnswerDto {
        +organizationId: string
        +questionnaireId?: string
        +questionsAndAnswers: AutoAnswerQuestionDto[]
    }

    class AutoAnswerQuestionDto {
        +question: string
        +answer?: string | null
        +_originalIndex?: number
    }

    class SaveAnswerDto {
        +questionnaireId: string
        +organizationId: string
        +questionAnswerId?: string
        +questionIndex?: number
        +answer?: string | null
        +status: 'manual' | 'generated'
        +sources?: any
    }

    class DeleteAnswerDto {
        +questionnaireId: string
        +organizationId: string
        +questionAnswerId: string
    }

    class UploadAndParseDto {
        +organizationId: string
        +fileName: string
        +fileType: string
        +fileData: string
        +source?: 'internal' | 'external'
    }

    class ExportByIdDto {
        +questionnaireId: string
        +organizationId: string
        +format: 'pdf' | 'csv' | 'xlsx'
    }

    class QuestionnaireAnswer {
        +question: string
        +answer: string | null
        +sources?: any
    }

    class ParsedQuestionnaireResult {
        +vendorName?: string
        +fileName?: string
        +totalQuestions: number
        +questionsAndAnswers: QuestionnaireAnswer[]
    }

    class QuestionnaireExportResult {
        +fileBuffer: Buffer
        +mimeType: string
        +filename: string
        +questionsAndAnswers: QuestionnaireAnswer[]
    }

    QuestionnaireController --|> QuestionnaireService : uses
    QuestionnaireService ..> ParseQuestionnaireDto
    QuestionnaireService ..> ExportQuestionnaireDto
    QuestionnaireService ..> AnswerSingleQuestionDto
    QuestionnaireService ..> SaveAnswerDto
    QuestionnaireService ..> DeleteAnswerDto
    QuestionnaireService ..> UploadAndParseDto
    QuestionnaireService ..> ExportByIdDto
    QuestionnaireService ..> QuestionnaireAnswer
    QuestionnaireService ..> ParsedQuestionnaireResult
    QuestionnaireService ..> QuestionnaireExportResult
    AutoAnswerDto o-- AutoAnswerQuestionDto : contains
\`\`\`
Sources:
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L34-L46)
[apps/api/src/questionnaire/questionnaire.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.controller.ts#L32-L36)
[apps/api/src/questionnaire/dto/parse-questionnaire.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/parse-questionnaire.dto.ts)
[apps/api/src/questionnaire/dto/export-questionnaire.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/export-questionnaire.dto.ts)
[apps/api/src/questionnaire/dto/answer-single-question.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/answer-single-question.dto.ts)
[apps/api/src/questionnaire/dto/auto-answer.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/auto-answer.dto.ts)
[apps/api/src/questionnaire/dto/save-answer.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/save-answer.dto.ts)
[apps/api/src/questionnaire/dto/delete-answer.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/delete-answer.dto.ts)
[apps/api/src/questionnaire/dto/upload-and-parse.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/upload-and-parse.dto.ts)
[apps/api/src/questionnaire/dto/export-by-id.dto.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/dto/export-by-id.dto.ts)

## Automatic Answering

The module provides robust capabilities for automatically generating answers to questionnaire questions using a Retrieval Augmented Generation (RAG) approach.

### Answer Generation Process

The `QuestionnaireService` orchestrates the automatic answering process:
1.  **Sync Organization Embeddings:** Before generating answers, the system ensures that the organization's knowledge base (policies, manual answers, etc.) is synchronized with the vector store. This step is crucial for accurate retrieval.
2.  **Batch Search:** For efficiency, questions are batched, and a single call to `findSimilarContentBatch` is made to retrieve relevant context from the vector store for all questions simultaneously.
3.  **Answer Generation:** For each question, `generateAnswerFromContent` is called, which uses the retrieved context and an AI model (e.g., `generateAnswerWithRAGBatch` in the service) to formulate an answer.
4.  **Save Answer:** Generated answers, along with their sources, are saved to the database.
5.  **Update Answered Count:** The total count of answered questions for the questionnaire is updated.

### Streaming Auto-Answer (SSE)

The `autoAnswer` endpoint in the `QuestionnaireController` provides a Server-Sent Events (SSE) stream to give real-time feedback on the answer generation progress.

<Steps>
<Step>
### Establish SSE Connection
The controller sets up SSE headers and creates a safe sender function to stream events back to the client.
</Step>
<Step>
### Sync Embeddings
The `QuestionnaireService` first attempts to synchronize the organization's embeddings to ensure the vector store is up-to-date. Warnings are logged if this fails, but the process continues.
</Step>
<Step>
### Filter Unanswered Questions
The incoming list of questions is filtered to identify those that still require an answer.
</Step>
<Step>
### Batch Context Search
A progress event (`type: 'progress'`, `phase: 'searching'`) is sent. The system then performs a batch search (`findSimilarContentBatch`) against the vector store to retrieve relevant content for all unanswered questions. This is a critical optimization to reduce latency.
</Step>
<Step>
### Parallel Answer Generation
Another progress event (`type: 'progress'`, `phase: 'generating'`) is sent. Answers are then generated in parallel for each question using the pre-fetched content (`generateAnswerFromContent`).
</Step>
<Step>
### Stream Individual Answers
As each answer is generated, an `answer` event is streamed back to the client, including the question, generated answer, sources, and success status.
</Step>
<Step>
### Save Generated Answers
Each generated answer is persisted to the database via `saveGeneratedAnswerPublic` in the `QuestionnaireService`.
</Step>
<Step>
### Complete Stream
Once all questions are processed, a `complete` event is sent, summarizing the total and answered questions. The SSE connection is then closed.
</Step>
</Steps>
Sources:
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L225-L269)
[apps/api/src/questionnaire/questionnaire.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.controller.ts#L368-L476)
[apps/api/src/vector-store/lib/index.ts](https://github.com/blade47/comp/blob/main/apps/api/src/vector-store/lib/index.ts)
[apps/api/src/trigger/questionnaire/answer-question-helpers.ts](https://github.com/blade47/comp/blob/main/apps/api/src/trigger/questionnaire/answer-question-helpers.ts)

\`\`\`mermaid
sequenceDiagram
    actor Client
    participant Controller as QuestionnaireController
    participant Service as QuestionnaireService
    participant VectorStore as Vector Store
    participant AI as AI Models
    participant DB as Database
    participant S3 as S3 Storage

    Client->>Controller: POST /auto-answer (AutoAnswerDto)
    Controller->>Controller: setupSSEHeaders()
    Controller->>Service: syncOrganizationEmbeddings(orgId)
    Service->>VectorStore: syncOrganizationEmbeddings(orgId)
    VectorStore-->>Service: Sync Status
    Service-->>Controller:
    Controller->>Client: SSE: progress (phase: searching)

    Controller->>VectorStore: findSimilarContentBatch(questions, orgId)
    VectorStore-->>Controller: allSimilarContent[]
    Controller->>Client: SSE: progress (phase: generating)

    loop For each question
        Controller->>AI: generateAnswerFromContent(question, similarContent)
        AI-->>Controller: AnswerResult {answer, sources}
        Controller->>Service: saveGeneratedAnswerPublic(questionnaireId, questionIndex, answer, sources)
        Service->>DB: Update questionnaireQuestionAnswer
        DB-->>Service:
        Service->>QuestionnaireStorage: updateAnsweredCount(questionnaireId)
        QuestionnaireStorage->>DB: Update questionnaire.answeredQuestions
        DB-->>QuestionnaireStorage:
        QuestionnaireStorage-->>Service:
        Service-->>Controller:
        Controller->>Client: SSE: answer {questionIndex, answer, sources}
    end

    Controller->>Client: SSE: complete {total, answered, answers}
    Controller->>Client: Close SSE connection
\`\`\`
Sources:
[apps/api/src/questionnaire/questionnaire.controller.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.controller.ts#L368-L476)
[apps/api/src/questionnaire/questionnaire.service.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/questionnaire.service.ts#L225-L269)
[apps/api/src/vector-store/lib/index.ts](https://github.com/blade47/comp/blob/main/apps/api/src/vector-store/lib/index.ts)
[apps/api/src/trigger/questionnaire/answer-question-helpers.ts](https://github.com/blade47/comp/blob/main/apps/api/src/trigger/questionnaire/answer-question-helpers.ts)
[apps/api/src/questionnaire/utils/questionnaire-storage.ts](https://github.com/blade47/comp/blob/main/apps/api/src/questionnaire/utils/questionnaire-storage.ts#L22-L36)

### DTOs for Auto-Answering

| DTO Class          | Description                                                                                                                                                                                                                         
\`\`\`

Was this page helpful?