The real power of a Schema Registry shows up when you need to change your data over time.
Without a registry, a schema change to a Kafka message requires coordinating simultaneous deployments across every producer and consumer. Get the timing wrong and you have an outage. With a registry and proper compatibility configuration, you can evolve schemas safely and incrementally.
The Compatibility Modes
Backward compatibility - new consumers can read data produced by old producers. In practice: deploy your updated consumers first, then update the producers at your own pace. No coordination required. This is the most commonly used mode.
Forward compatibility - old consumers can still read data produced by new producers. In practice: deploy your updated producers first, then update consumers when ready. Useful when the producer change is urgent and consumer updates can follow.
Full compatibility - both backward and forward compatibility are maintained simultaneously. Either side can be updated in any order without affecting the other. The most restrictive to maintain but the most operationally flexible.
What the Registry Enforces
When you submit a new schema version, the registry validates it against the configured compatibility mode before accepting it. A change that violates compatibility is rejected at registration time - before it ever reaches production, before any consumer can receive a message it cannot parse.
In practice, this means:
- Adding an optional field with a default value: allowed under backward and full compatibility
- Removing a field: allowed only under forward compatibility (old consumers ignore unknown fields)
- Changing a field's type: almost never compatible, rejected under all strict modes
- Adding a new enum value: allowed under forward compatibility with default handling
The PolicyState interface is a good example of a schema that evolves. It has seven behavioral dimensions that inform every retrieval and action decision in the system:
// packages/core-types/src/policy.ts
export interface PolicyState {
readonly version: string
readonly timestamp: string
/** Bias toward memory retrieval over in-weights inference. */
readonly retrievalBias: number
/** Preference for tool invocation over pure reasoning. */
readonly toolBias: number
/** Tolerance for uncertain or risky actions. */
readonly riskTolerance: number
/** Degree of trust placed in retrieved memories. */
readonly memoryTrust: number
/** Propensity to explore novel strategies rather than exploit known ones. */
readonly explorationFactor: number
/** Weight given to long-horizon goals over immediate task completion. */
readonly goalPersistence: number
/** How aggressively working memory is pruned between turns. */
readonly workingMemoryDecayRate: number
}Every time a new behavioral dimension is added to PolicyState, every consumer reading policy updates needs to handle it. Without schema evolution compatibility enforcement, adding workingMemoryDecayRate to this interface is a quiet breaking change for any consumer that was not updated simultaneously. With backward compatibility mode, the new field gets a default value and old consumers continue reading without modification.
The Practical Impact
For the Cognitive Substrate, schema compatibility has been essential. The ExperienceEvent schema has gone through multiple iterations as I learned what data actually matters. Not one of those changes required a coordinated deployment window or caused a pipeline outage.
The changes were safe because the registry enforced the compatibility rules before the changes could reach production.