#kafka

KIP-1150 introduces a new type of Kafka topic that stores data directly in cloud object storage instead of on broker disks - eliminating cross-AZ replication costs and dramatically simplifying broker scaling. Here is how it works and why Aiven built Inkless to implement it early.

It is tempting to skip the Schema Registry when you are moving fast. Here is what that decision actually costs - in debugging time, fragile consumer code, onboarding difficulty, and silent data corruptions.

How I actually use the Schema Registry in the Cognitive Substrate - Confluent Registry with Avro schemas, CI-enforced compatibility checks, schema-ID-based consumption, and the specific practices that made the pipeline reliable.

The real power of a Schema Registry shows up when you need to change your message formats over time. Backward, forward, and full compatibility modes - and how to evolve schemas safely without coordinating simultaneous deployments.

A Schema Registry is a central repository for the structure of every message type flowing through Kafka. It enforces contracts, enables safe schema evolution, and turns your event stream into living documentation.

Using Kafka without a schema registry sounds fine in theory - just send JSON. In practice, it becomes a debugging nightmare. How one renamed field broke the ingestion pipeline and why I stopped trusting raw messages.

The Cognitive Substrate needs to ingest a constant stream of events from many different sources - telemetry, tickets, Slack, logs. Here is why Kafka is the right foundation for that, and what I quickly learned it was not enough on its own.

This article records the first hosted experiment in which Cognitive Substrate converted live infrastructure telemetry into embedded operational memory and used that memory inside the normal workbench

How operational knowledge learned in one infrastructure environment transfers to another: the system-mapping boundary, zero-shot pattern application, local confidence calibration, and what cannot transfer.

This article describes the worker that detects operational failure patterns from streams of operational primitive events and emits recommendations.

The telemetry ingestion worker: how raw infrastructure metrics are persisted to ClickHouse and translated into operational primitive events, with intentional discard semantics and dual Kafka output streams.

The ClickHouse telemetry layer for the operational intelligence pipeline: schema design for raw hot-tier and cognitive-tier tables, time-based partitioning, typed worker integration, and separation of raw from cognitive stores.

The operational primitive taxonomy: a closed, system-agnostic vocabulary that maps vendor telemetry from Kafka, OpenSearch, PostgreSQL, and ClickHouse into portable pattern signatures for cross-environment operational intelligence.

This article describes the integration of specialized agents into a coordinated runtime that can scale across distributed infrastructure.

The reinforcement layer turns outcome evidence into structured scoring signals for memory priority, policy evaluation, and identity-impact records.

The policy engine provides bounded behavioral drift, converting evaluated outcomes into clamped policy-vector updates and emitting inspectable adaptation records.

A reading of Kafka not as a message queue but as an episodic memory substrate - ordered, immutable, queryable across time.

Consolidation gives memory an offline replay path, selecting replay candidates, building semantic drafts, and emitting update events in a sleep-cycle-like architecture.

This article opens the public series on Cognitive Substrate: how persistent, learnable memory differs from logging, and how ingestion turns structured experience into durable archive plus searchable index.

How to model user attention and comprehension as typed events, then route them through a distributed pipeline to ClickHouse and OpenSearch.