Orientation
Content Processing
Search Infrastructure
Rendering and UI
Developer Tooling
How It Works
The following files were used as context for generating this wiki page:
Search Indexing is the engine that enables performant full-text search within documentation sites. By transforming raw content—typically Markdown or MDX—into a structured, searchable format, the indexing system bridges the gap between static documents and dynamic user queries. It solves the performance bottleneck of scanning large text bodies in real-time by generating searchable indices (whether static, in-memory, or hosted by third-party services) during the build process.
The system is architected as a modular pipeline: a "content extractor" (such as remark-structure) traverses the AST to collect headings and paragraphs, which are then passed to a "builder" component that converts these into documents compatible with specific search backends (Orama, FlexSearch, or Algolia). This decoupling ensures that documentation authors can swap indexing strategies or backends without rewriting their content source logic.
Crucially, the indexing lifecycle interacts closely with the core loader system, which manages the virtual file system and page hierarchy. By centralizing index building, the system guarantees that metadata like breadcrumbs, unique page IDs, and language-specific translations remain consistent across both the rendering layer and the search interface.
remark-structureThe remark-structure plugin acts as the primary content extractor. During the MDX build process, it transforms raw files into StructuredData, a data model consisting of headings and contents arrays.
When remarkStructure traverses the MDX tree, it uses a configurable stringify function to pull text out of elements. It relies on a lastHeading pointer to associate paragraphs with the closest preceding header, creating a flat structure that links text content to its specific context.
Tip
Always use remark-heading before remark-structure. If remark-structure finds a heading missing an hProperties.id, it will skip that heading, preventing it from being added to the StructuredData output.
Once data is extracted, it must be normalized into a backend-agnostic document format. The buildDocuments function performs this transformation. It takes a list of SharedIndex objects—each containing metadata like titles, URLs, and structured data—and produces an array of SharedDocument entities.
The indexing logic iterates through headings and contents, assigning each a unique ID using a counter and creating granular search entries. The nextId counter ensures that every individual paragraph or heading has a distinct anchor.
Orama is an in-memory search engine supported natively by the system. The createDBSimple and createDB (advanced) functions act as the bridge to Orama's API.
For simple search, createDBSimple maps the SharedIndex into an object matching simpleSchema and calls insertMultiple to populate the Orama DB.
FlexSearch, an alternative in-memory backend, is managed through flexsearch.ts. It wraps the library's Document constructor and exposes an export method that serializes the state to a JSON-compatible object.
The system uses a createDocument factory that enforces a specific schema, requiring an id, content field for indexing, and a tags array for filtering. Unlike Orama, FlexSearch uses index.add inside an initIndex routine.
For larger, production-scale deployments, the system integrates with Orama Cloud via packages/core/src/search/orama-cloud.ts. This involves a push-based model where local indexes are transformed into OramaDocument types and synced using a transaction API.
Important
The sync process uses a transaction: index.transaction.open() starts it, insertDocuments adds the batch, and index.transaction.commit() finalizes the update. If autoDeploy is true, the snapshot is deployed to the production index automatically.
The client-side search functionality depends on the backend chosen. For static clients (oramaStaticClient or flexsearchStaticClient), the client fetches a serialized index file.
fetch request to an API endpoint (e.g., /api/search).load (for Orama) or import (for FlexSearch).Map to prevent redundant fetches.Algolia support is handled through a sync function in packages/core/src/search/algolia.ts. It performs an index replacement: it sets index settings (like searchable attributes and faceting) and then calls client.replaceAllObjects to ensure the cloud index perfectly matches the local source state.