Operating the Substrate in Production

Building a system is one thing. Running it in production, day after day, is something else entirely.

The Monitoring Problem

Standard service monitoring tells you whether the system is up and whether it is responding quickly. That is necessary but not sufficient for a memory system.

You also need to observe memory quality - and that requires custom instrumentation:

Consolidation lag - how far behind is the consolidation worker from the current event watermark? A growing lag means raw experiences are accumulating faster than they are being processed.

Trust score distribution - are trust scores distributing across the full range, or clustering at the top (everything looks high quality) or bottom (the feedback loop is not working)?

Retrieval quality stability - is the average trust score of retrieved memories stable over time, or slowly declining as the memory store grows?

Memory tier distribution - are memories moving between tiers as expected, or is everything stacking up in one tier?

Every service in the Cognitive Substrate calls initTelemetry() at startup to register traces, metrics, and OTLP export. The OTel setup is environment-driven and includes a clean shutdown path for flushing pending spans:

// packages/telemetry-otel/src/bootstrap.ts
//
// Call initTelemetry() once at startup - before any imports that use the OTel API.
// Config from environment:
//   OTEL_SERVICE_NAME            - required
//   OTEL_EXPORTER_OTLP_ENDPOINT  - optional (e.g. http://collector:4318)
//   OTEL_SDK_DISABLED            - "true" to disable in unit tests

export async function initTelemetry(config: TelemetryConfig): Promise<() => Promise<void>> {
  if (config.disabled) {
    return async () => undefined
  }

  const resource = new Resource({
    [SEMRESATTRS_SERVICE_NAME]: config.serviceName,
    [SEMRESATTRS_SERVICE_VERSION]: config.serviceVersion ?? '0.0.0',
  })

  const traceExporter = config.otlpEndpoint
    ? new OTLPTraceExporter({ url: `${config.otlpEndpoint}/v1/traces` })
    : new OTLPTraceExporter()

  _sdk = new NodeSDK({ resource, traceExporter })
  _sdk.start()

  // Return a shutdown function - caller must invoke on SIGTERM to flush spans.
  return async () => {
    await _sdk?.shutdown()
  }
}

// Each service builds its config from env vars:
export function telemetryConfigFromEnv(defaultServiceName: string): TelemetryConfig {
  return {
    serviceName: process.env['OTEL_SERVICE_NAME'] ?? defaultServiceName,
    serviceVersion: process.env['npm_package_version'] ?? '0.0.0',
    disabled: process.env['OTEL_SDK_DISABLED'] === 'true',
    otlpEndpoint: process.env['OTEL_EXPORTER_OTLP_ENDPOINT'],
  }
}

W3C traceparent headers are injected into every Kafka message by CognitiveProducer and extracted by CognitiveConsumer. This means distributed traces span the full pipeline - from telemetry ingestion through consolidation to retrieval - without any manual instrumentation in the worker code.

Schema Evolution

The ExperienceEvent schema has changed multiple times as I learned more about what data matters. Each schema change requires careful handling: new documents use the new format, old documents use the old format, and retrieval needs to work correctly for both during the migration window.

The solution is versioned document formats in OpenSearch and a retrieval layer that normalizes between versions at query time. Schema migrations run as background jobs that gradually rewrite old documents to the current format.

The Embedding Model Problem

The most expensive operational surprise: embedding model drift.

When I updated the embedding model used during ingestion, historical memories became semantically incompatible with new queries. The vector spaces of the old and new models are not aligned - a query embedded with the new model does not retrieve memories embedded with the old model reliably.

Fixing this requires re-embedding the entire historical corpus with the new model. That is a significant compute job that requires careful coordination with the live retrieval path to avoid serving degraded results during the transition.

The lesson: treat the choice of embedding model as a high-stakes long-term commitment, not an implementation detail.

What Production Actually Requires

Having a working system is great. Having a system you can trust in production - with alerts that fire when something actually matters, migrations that can run without downtime, and enough observability to diagnose problems without guessing - is the real engineering goal.

This is the final piece of infrastructure that turns the Cognitive Substrate from a promising project into something you can actually depend on.