Skip to main content
doc0blade47/mem0documentation home
Sign in

Entity Extraction

Last updated Oct 11, 2026100% grounded
Relevant source files

The following files were used as context for generating this wiki page:

Overview

Entity Extraction powers the automated parsing and linking of structured knowledge units from unstructured conversation text and agent interaction episodes. By combining natural language processing models, regex-based heuristic filtering, and LLM-driven fact prompting, the module isolates significant proper nouns, quoted strings, technical identifiers, and multi-word topic phrases while discarding irrelevant filler.

Sources: mem0/utils/entity_extraction.py:1-16

This subsystem bridges raw conversational streams with semantic vector storage by deduplicating candidate spans, resolving overlapping entity boundaries, and attaching normalized records to persistent memory IDs across user and agent scopes.

Sources: mem0/memory/main.py:1100-1115

Natural Language Model Loading

Shared Model Initialization

The engine consolidates natural language processing initialization through mem0/utils/spacy_models.py, which shares model instances between entity extraction and BM25 text processing. Two distinct pipeline configurations are maintained globally: _nlp_full for full pipeline tasks including named entity recognition and dependency parsing, and _nlp_lemma with disabled components for lightweight lemmatization.

Sources: mem0/utils/spacy_models.py:1-15

Thread safety during model acquisition is enforced using a module-level threading.Lock(). If model loading fails once, subsequent calls short-circuit by returning None immediately via _load_failed_full and _load_failed_lemma flags without re-entering the locked block.

Sources: mem0/utils/spacy_models.py:16-19, mem0/utils/spacy_models.py:44-55, mem0/utils/spacy_models.py:69-80

Note

The spaCy model en_core_web_sm is automatically acquired via spacy.cli.download if spacy.util.is_package detects it missing, throwing an ImportError if spaCy itself is absent or a RuntimeError if download fails. Sources: mem0/utils/spacy_models.py:21-42

Pipeline Configuration Reference

FunctionPipeline ComponentsDisabled ComponentsTarget Use Case
get_nlp_full()Default full pipelineNoneEntity extraction (_extract_entities_from_doc)
get_nlp_lemma()Tokenizer, tagger, morpher, lemmatizer["ner", "parser"]BM25 text processing and lemma lookups

Sources: mem0/utils/spacy_models.py:44-66, mem0/utils/spacy_models.py:69-91

Batch Processing and Fallbacks

Entity extraction workflows consume the shared loader directly within extract_entities and extract_entities_batch. When processing multiple texts, extract_entities_batch leverages spaCy's nlp.pipe iterator with a configurable batch_size default of 32 items. If the spaCy package is missing or model loading encounters an unrecoverable exception, both functions return empty lists ([] or [[] for _ in texts]) to let higher-level extraction pipelines proceed gracefully without crashing.

Sources: mem0/utils/entity_extraction.py:751-772, mem0/utils/spacy_models.py:56-66, mem0/utils/spacy_models.py:81-91

Caution

Disabling components like ner and parser via disable=["ner", "parser"] in get_nlp_lemma() makes that instance unsuitable for dependency parsing or named entity recognition; attempting to extract entity candidates with it will yield empty results. Sources: mem0/utils/spacy_models.py:69-91

Candidate Entity Extraction Pipeline

Entity Extraction Phases and Candidate Types

The entity extraction engine parses a processed spaCy document through five specialized candidate generators. Each generator appends typed candidates into a shared _EntityCandidate dataclass collection, tracking priority, source type, character offsets, and confidence metrics before resolution.

Sources: mem0/utils/entity_extraction.py:24-33, mem0/utils/entity_extraction.py:731-748

The engine yields four distinct entity types categorized during parsing: PROPER, QUOTED, TOPIC, and IDENTIFIER.

Sources: mem0/utils/entity_extraction.py:13-15

Recognized Entity Labels and Filters

Named Entity Recognition (NER) candidate extraction relies on explicit lists of accepted and rejected spaCy entity labels. Accepted labels cover standard named entities while excluding numerical, temporal, and monetary categories.

Label CategoryIncluded Entity Labels
_ACCEPTED_NER_LABELSPERSON, ORG, GPE, LOC, FAC, PRODUCT, WORK_OF_ART, EVENT, NORP, LAW, LANGUAGE
_REJECTED_NER_LABELSDATE, TIME, CARDINAL, ORDINAL, QUANTITY, MONEY, PERCENT

Sources: mem0/utils/entity_extraction.py:98-122

Note

Technical identifiers are matched using a regular expression pattern requiring a letter or underscore, followed by word characters or hyphens, separated by at least one dot component ([A-Za-z_][\w-]*(?:\.[A-Za-z_][\w-]*)+). Sources: mem0/utils/entity_extraction.py:404-406

Filtering Terminology and Formatting Markers

To prevent generic terms from polluting memory records, the pipeline consults multiple suppression sets. Single-word proper names, noun compound heads, and trailing words are filtered against predefined exclusion lists.

Suppression SetSample EntriesPurpose
_GENERIC_HEADSthing, stuff, way, time, experience, situation, case, factPrevents generic nouns from serving as entity heads
_GENERIC_SINGLE_ENTITY_TERMSuser, assistant, agent, customer, client, memory, chatRejects common role words and single-token pseudo-entities
_CIRCUMSTANTIAL_MODSsolo, individual, team, group, joint, first, last, mainFilters modifiers describing circumstance rather than content
_GENERIC_ENDINGSwork, job, task, info, data, content, results, itemStrips generic trailing words from compound token sequences
_FORMATTING_MARKERS*, -, +, •, –, —, #, ##, ###, **, __Markdown and formatting symbols skipped during extraction

Sources: mem0/utils/entity_extraction.py:35-96, mem0/utils/entity_extraction.py:124-162, mem0/utils/entity_extraction.py:283-317, mem0/utils/entity_extraction.py:353

Candidate Filtering and Heuristic Resolution

Conflict Resolution and Deduplication Flow

The engine resolves overlapping and duplicate entity candidates inside the _resolve_candidates routine. Candidates are first normalized by their text representation and deduplicated by selecting the entry with the highest priority and confidence score. Surviving candidates are then sorted and scanned for overlapping character spans, filtering out redundant matches unless a specific multi-word topic spans across a proper noun boundary.

Sources: mem0/utils/entity_extraction.py:708-728

Note

During span overlap resolution, multi-word topic phrases (TOPIC with spaces) are permitted to coexist with overlapping proper noun entities (PROPER), whereas other overlapping span conflicts drop the lower-priority candidate. Sources: mem0/utils/entity_extraction.py:719-724

Sorting and Acceptance Criteria

Sorting keys applied during candidate resolution combine four distinct prioritization attributes. The sorting tuple (c.priority, -c.confidence, -(c.end - c.start), c.start) ensures that higher-priority, higher-confidence, and longer span matches take precedence during the linear overlap check.

Accepted candidates are subsequently resorted by their character start positions before output tuple generation.

source
mem0/utils/entity_extraction.py:713-727

Sources: mem0/utils/entity_extraction.py:713-727

Warning

Candidates with negative or unassigned start offsets default to index 10**9 during final position sorting, placing them at the end of the extracted sequence. Sources: mem0/utils/entity_extraction.py:727

Prompt Orchestration for Memory Extraction

Additive Extraction Prompt Architecture

The TypeScript OSS package orchestrates prompt generation for memory extraction through the generateAdditiveExtractionPrompt builder and static prompt definitions in mem0-ts/src/oss/src/prompts/index.ts. The orchestration framework coordinates context sections including conversation summaries, past messages, existing memories, new conversational input, and temporal anchor points.

Sources: mem0-ts/src/oss/src/prompts/index.ts:822-863

Note

The builder function generateAdditiveExtractionPrompt defaults both currentDate and observationDate to today's ISO date string (YYYY-MM-DD) derived from new Date().toISOString().split("T")[0] if they are not explicitly supplied in the options parameter. Sources: mem0-ts/src/oss/src/prompts/index.ts:830-832

Conversation History Serialization and Truncation

Prior message arrays are serialized via formatConversationHistory, which iterates through message objects, extracts role and content attributes, and truncates individual messages exceeding PAST_MESSAGE_TRUNCATION_LIMIT characters.

source
mem0-ts/src/oss/src/prompts/index.ts:791-820

Sources: mem0-ts/src/oss/src/prompts/index.ts:791-820

Warning

The truncation constant PAST_MESSAGE_TRUNCATION_LIMIT is hardcoded to 300 characters; any past message content longer than 300 characters is sliced and appended with an ellipsis (...) before being formatted into the ## Last k Messages section. Sources: mem0-ts/src/oss/src/prompts/index.ts:791-799

Agent Session Episode Assembly

Episode Structure and Semantic Evidence

Coding session episodes are assembled into structured text content that incorporates task metadata, file modification lists, file read lists, observed tool outputs, and repository provenance identifiers. The episode assembly routine formats tasks, paths, commands, searches, conclusions, and outcomes into a bounded string format. Sources: integrations/mem0-agent-plugin/core/memory_core.py:1619-1654

Semantic evidence extraction processes modified paths and observed commands from the structured session data. Test and build results are deliberately filtered out of semantic evidence so that transient verification details do not steer memory extraction away from core repository knowledge. Sources: integrations/mem0-agent-plugin/core/memory_core.py:1657-1673

ComponentContent FilterBound LimitPurpose
taskRaw stringNoneIdentifies the objective of the session
modified_pathsSliced to first 50 items500 chars per pathRecords changed files in the repository
read_pathsSliced to first 50 itemsNoneRecords inspected code files
commandsSliced to last 30 items, status checkNoneCaptures observed execution commands
searchesSliced to last 20 items, JSON dumpNoneCaptures query patterns and file searches
conclusionRaw stringNoneSummarizes the final state of the task
outcomeRaw stringNoneRecords success or failure description

Sources: integrations/mem0-agent-plugin/core/memory_core.py:1620-1654

Note

Command logs within build_semantic_evidence are filtered to retain only entries where the command status equals failed, ensuring routine passing test executions are pruned from persistent semantic memory. Sources: integrations/mem0-agent-plugin/core/memory_core.py:1667-1669

Message Batching and Token Budget Enforcement

Session extraction messages are filtered to retain only user and assistant roles with non-empty content, after which semantic evidence is appended to the final assistant message or added as a new assistant record. Sources: integrations/mem0-agent-plugin/core/memory_core.py:1688-1704

Tokens are conservatively estimated via _estimated_tokens, which computes an ASCII-weighted byte and character length ratio without introducing a heavy tokenizer dependency. Sources: integrations/mem0-agent-plugin/core/memory_core.py:1707-1710

Sources: integrations/mem0-agent-plugin/core/memory_core.py:1735-1798

Warning

Subagent interactions identified by _is_agent_assignment and _is_agent_response are explicitly kept paired together during token budget splitting to prevent splitting context across extraction batch boundaries. Sources: integrations/mem0-agent-plugin/core/memory_core.py:1717-1727, 1756-1762

Related

Was this page helpful?
Powered by doc0