KIP-1150 is one of the most significant proposed changes to Apache Kafka in years - and Aiven built an open-source implementation called Inkless before it landed in mainline Kafka.
What KIP-1150 Proposes
KIP-1150: Diskless Topics introduces a new type of Kafka topic that writes partition data directly to cloud object storage (S3, GCS, or equivalent) instead of to broker-local disks.
In traditional Kafka, data flows like this: producer writes to the leader broker's local disk, the leader replicates to follower brokers in other availability zones, consumers read from brokers. Each piece of data crosses availability zone boundaries at least once, often multiple times.
With diskless topics, the durable copy of the data lives in object storage. Brokers act primarily as a routing and caching layer - they receive writes, buffer them briefly in memory or a small local cache, then flush to object storage. Consumers read from brokers, which serve data from their cache when available and from object storage when not.
The Cost Motivation
Cross-AZ data transfer is one of the more expensive line items in a cloud Kafka deployment. Traditional Kafka replication requires crossing AZ boundaries to maintain durability across availability zones. In high-throughput deployments, this cost can dominate the total bill.
Diskless topics eliminate this by delegating durability to object storage, which handles replication internally and does not charge for cross-AZ transfers within its own replication. The broker-to-broker replication traffic goes to near zero.
Reports from early adopters suggest total cost of ownership reductions of up to 80% for appropriate workloads.
Inkless: Aiven's Implementation
Aiven runs managed Kafka at scale and encountered the cross-AZ cost problem directly. Rather than waiting for KIP-1150 to be accepted and land in mainline Kafka, they built Inkless - an open-source fork that implements the KIP-1150 storage model.
Inkless is intended as a temporary fork - the goal is for the changes to be incorporated into mainline Kafka through the KIP process, at which point Inkless would be retired. But for organizations that need the cost benefits now, Inkless provides a production-usable implementation.
The Tradeoffs
Diskless topics are not universally better than traditional topics.
Higher latency - object storage writes are slower than local disk writes. End-to-end produce latency increases. For workloads that need consistent sub-millisecond latency, diskless topics are not appropriate.
Object storage dependency - availability and performance become coupled to the object storage service. S3 is highly reliable, but it is a different failure mode than local disk.
Cost structure changes - the savings on cross-AZ transfer come with increased object storage API request costs. The net economics are favorable for high-throughput workloads but need to be evaluated for each specific case.
Relevance for the Cognitive Substrate
The Cognitive Substrate's event ingestion pipeline is a high-throughput, cost-sensitive workload where sub-millisecond latency is not required. Events can tolerate a few hundred milliseconds of additional latency through the Kafka layer without affecting memory quality.
This makes it a reasonable candidate for diskless topics. Reducing the cost of the event ingestion infrastructure directly reduces the cost of running the full Cognitive Substrate at scale.
The topic configuration in the Cognitive Substrate already designates the telemetry and cognition tiers as diskless. The diskless: true flag is set at topic declaration time, separating storage policy from the rest of the application logic:
// packages/kafka-bus/src/topics.ts
export const TOPIC_CONFIGS: ReadonlyArray<TopicConfig> = [
// Standard Kafka topics - broker-local storage, low latency required.
{
name: Topics.EXPERIENCE_RAW,
partitions: 12,
retentionMs: 7 * 86_400_000,
replicationFactor: 3,
},
{
name: Topics.EXPERIENCE_ENRICHED,
partitions: 12,
retentionMs: 7 * 86_400_000,
replicationFactor: 3,
},
{
name: Topics.MEMORY_INDEXED,
partitions: 6,
retentionMs: 30 * 86_400_000,
replicationFactor: 3,
},
{
name: Topics.AGENT_REASONING_REQUEST,
partitions: 12,
retentionMs: 3 * 86_400_000,
replicationFactor: 3,
},
{
name: Topics.AGENT_REASONING_RESPONSE,
partitions: 12,
retentionMs: 3 * 86_400_000,
replicationFactor: 3,
},
{
name: Topics.CONSOLIDATION_REQUEST,
partitions: 6,
retentionMs: 3 * 86_400_000,
replicationFactor: 3,
},
{
name: Topics.POLICY_EVALUATION,
partitions: 3,
retentionMs: 7 * 86_400_000,
replicationFactor: 3,
},
{ name: Topics.AUDIT_EVENTS, partitions: 6, retentionMs: 90 * 86_400_000, replicationFactor: 3 },
// Telemetry tier - KIP-1150 Diskless (object storage).
// High-volume append-only; 500ms-5s broker latency acceptable.
{
name: Topics.TELEMETRY_METRICS_RAW,
partitions: 24,
retentionMs: 7 * 86_400_000,
replicationFactor: 3,
diskless: true,
},
{
name: Topics.TELEMETRY_LOGS_RAW,
partitions: 24,
retentionMs: 7 * 86_400_000,
replicationFactor: 3,
diskless: true,
},
{
name: Topics.TELEMETRY_TRACES_RAW,
partitions: 24,
retentionMs: 7 * 86_400_000,
replicationFactor: 3,
diskless: true,
},
// Cognition tier - KIP-1150 Diskless (object storage).
// Processed operational intelligence; minutes-level latency acceptable.
{
name: Topics.COGNITION_PRIMITIVES,
partitions: 12,
retentionMs: 14 * 86_400_000,
replicationFactor: 3,
diskless: true,
},
{
name: Topics.COGNITION_PATTERNS,
partitions: 6,
retentionMs: 30 * 86_400_000,
replicationFactor: 3,
diskless: true,
},
{
name: Topics.COGNITION_ANOMALIES,
partitions: 6,
retentionMs: 90 * 86_400_000,
replicationFactor: 3,
diskless: true,
},
]The partition counts reflect the throughput difference: telemetry topics get 24 partitions to handle raw event volume, while standard pipeline topics need only 6-12. The diskless designation is only applied where the latency tradeoff is acceptable - the core pipeline topics that feed real-time reasoning remain on standard broker storage.