Avro
A binary serialization format used with the Schema Registry to encode Kafka messages compactly and with schema enforcement. Avro messages include a schema ID rather than the full schema, keeping message payloads small while still allowing consumers to retrieve and validate the full structure.
Apache Avro is the serialization format of choice for the Cognitive Substrate's Kafka pipeline when schema enforcement and compact encoding matter. Rather than repeating the full JSON structure in every message, an Avro-encoded message embeds only a numeric schema ID (registered in the Schema Registry) plus the binary-encoded payload. Consumers resolve the schema from the registry at startup and use it for deserialization. This combination - compact wire format plus registry-enforced contracts - eliminates the class of silent corruption bugs that arise when JSON field names or types drift between producers and consumers. Avro also supports schema evolution within compatibility rules, allowing fields to be added or deprecated without breaking existing consumers.