The Retrieval Bottleneck

As the memory store grows, the first problem is not storage - it is retrieval.

Why Naive Vector Search Does Not Scale

Semantic search works by computing the distance between your query vector and every stored vector, then returning the closest matches. This is called exact nearest neighbor search, and it scales linearly with the number of stored vectors.

At 50,000 memories, it is fast. At 500,000, it starts to feel slow. At 5 million, running this computation on every incoming question is both too slow and too expensive to sustain.

The arbitration layer makes this worse. It needs multiple candidates to score and rank - not just one result, but potentially hundreds. The more memories in the store, the more candidates retrieval has to generate before arbitration can do its job.

Here is the actual retrieval pipeline. The key design is the over-fetch pattern: when a reranker is configured, the system fetches 3x the requested number of candidates so the cross-encoder has a rich enough pool to meaningfully improve precision:

// packages/retrieval-engine/src/retriever.ts
//
// Three-stage pipeline:
//   1. Resolve query embedding (pre-computed or fresh from embedder)
//   2. Issue hybrid (BM25 + k-NN) queries per index in parallel
//   3. Optionally re-score with a Tier-2 cross-encoder and slice to final size

async retrieve(request: RetrievalRequest): Promise<RetrievalResult> {
  const queryEmbedding = await this.resolveEmbedding(request);
  const indexes = request.indexes ?? ["memory_semantic", "experience_events"];
  const finalSize = request.size ?? 10;

  // Over-fetch when a reranker is configured so the cross-encoder
  // has enough candidates to choose from.
  const perIndexSize = this.reranker
    ? (request.perIndexSize ?? finalSize) * this.rerankOverfetchFactor  // default 3x
    : (request.perIndexSize ?? finalSize);

  const hitsByIndex = await Promise.all(
    indexes.map(async (index) => {
      const query = buildHybridQuery({
        queryText: request.queryText,
        queryEmbedding,
        size: perIndexSize,
        k: perIndexSize,
        ...queryOptionsForIndex(index),
      });
      return { index, hits: await this.searchClient(this.openSearch, index, query) };
    }),
  );

  const candidates = hitsByIndex.flatMap(({ index, hits }) =>
    hits.map((hit) => mapSearchHitToMemoryReference(index, hit)),
  );

  if (this.reranker && candidates.length > 0) {
    return { memories: await this.applyReranking(request.queryText, candidates, finalSize), queryEmbedding };
  }

  return {
    memories: candidates.sort((l, r) => r.score - l.score).slice(0, finalSize),
    queryEmbedding,
  };
}

The HNSW index configuration - Lucene engine, m: 16, ef_construction: 128, ef_search: 100 - is what enables approximate nearest neighbor search across millions of vectors. Exact k-NN is not used in production.

The Techniques That Help

Approximate Nearest Neighbor (ANN) indexing - instead of computing exact distances to every vector, ANN algorithms use indexing structures (HNSW is the most common) that find approximate nearest neighbors much faster. The tradeoff is a small reduction in recall accuracy that is acceptable in most practical cases.

Result caching - many questions in an operational domain are variations of the same few queries. Caching results for common patterns eliminates most of the retrieval overhead for these frequent cases.

Metadata pre-filtering - before running any vector operations, filter the candidate set using structured metadata: source type, time range, minimum trust score, domain tag. This reduces the search space dramatically before the expensive computation runs.

Query routing - not every question needs full semantic search. Many can be resolved with precise metadata filters alone, with vector search as a fallback only when structured filters return insufficient results. Building this routing logic into the retrieval layer pays significant dividends at scale.

Where Things Stand

The transition is from exact KNN to HNSW-indexed approximate search in OpenSearch, adding a result cache for the most common query patterns, and implementing metadata pre-filtering as the first stage of every retrieval request.

Retrieval speed is the hidden killer of most memory systems. Getting this right early is much easier than fixing it after the system has grown.