The Registry in Practice: My Implementation

Now that the theory is covered, here is how I actually use the Schema Registry in the Cognitive Substrate.

The Setup

I use Confluent Schema Registry alongside the Kafka cluster. Every ExperienceEvent has a formal Avro schema registered in the registry. Before any new event type can be published to a Kafka topic, its schema must be registered and validated for compatibility with any existing versions.

Avro was the right format choice: compact binary serialization, native support in Confluent's ecosystem, and excellent tooling for compatibility checking.

Consumers Bind to Schema IDs

One of the most important practices adopted: consumers subscribe to schema IDs rather than relying on implicit knowledge of the current topic format.

When the ingestion worker processes a message, it reads the schema ID embedded in the message header, retrieves the schema definition from the registry (cached after the first fetch), and deserializes using that exact schema. This means multiple schema versions can coexist on the same Kafka topic - perhaps during a rolling deployment - without any consumer-side branching logic.

The CognitiveConsumer.subscribe method handles this transparently. W3C traceparent headers are extracted alongside the payload so distributed traces span the full pipeline without manual instrumentation in worker code:

// packages/kafka-bus/src/consumer.ts

async subscribe<T>(
  topics: ReadonlyArray<TopicName>,
  handler: MessageHandler<T>,
): Promise<void> {
  for (const topic of topics) {
    await this.consumer.subscribe({ topic, fromBeginning: false });
  }
  await this.consumer.run({
    eachMessage: async (payload: EachMessagePayload): Promise<void> => {
      const { topic, partition, message } = payload;
      if (!message.value) return;
      const value = JSON.parse(message.value.toString()) as T;
      // Extract W3C traceparent headers for distributed tracing continuity.
      const traceContext = extractTraceContext(message.headers ?? undefined);
      await handler({
        topic, partition,
        offset: message.offset,
        timestamp: message.timestamp,
        key,
        value,
        traceContext,
      });
    },
  });
}

The schema ID is embedded by the producer client at publish time, so consumers never need to guess or hard-code an expected format. When a new schema version is registered, existing consumers continue deserializing messages with their known schema ID while new messages carry the updated one.

CI Compatibility Checks

This is the practice that made the biggest difference: schema changes go through the same pull request review process as code changes, and the CI pipeline validates proposed schema changes against the registry's configured compatibility rules before the PR can be merged.

A breaking schema change fails the CI build, not the production pipeline. The problem is caught at the earliest possible point, by the developer who introduced it, before any other system is affected.

Version History

A clean version history is maintained for every event type in the registry. This has proven useful for debugging historical processing issues (correlating when a schema changed with when a pipeline anomaly appeared) and for onboarding contributors who need to understand how the event contracts have evolved.

The Result

Schema-related failures have essentially disappeared from the Kafka pipeline. When something breaks now, I look elsewhere first - because unexpected message format issues simply do not reach production anymore.

This single architectural decision has given me significantly more confidence in the entire event ingestion system.