Diskless topic
A Kafka topic backed by object storage rather than broker-local disks. Brokers become stateless routing nodes. Suitable for high-volume append-only streams where object-storage latency is acceptable and long retention is required at low cost.
Traditional Kafka brokers store topic data on local attached disks. When a broker fails, its partitions must be recovered from replicas on other brokers, which requires extra replication overhead and careful disk capacity management. Diskless topics (specified in KIP-1150 and implemented as Inkless Kafka) replace broker-local storage with object storage: the broker routes writes to S3 or equivalent and reads back from there, holding no durable state itself. This makes brokers stateless - a failed broker can be replaced instantly without data recovery, and the partition data persists in object storage regardless of broker lifecycle. The tradeoff is latency: object storage writes are slower than local disk writes, so diskless topics are only suitable for streams where consumers tolerate seconds of end-to-end latency rather than milliseconds. In the Cognitive Substrate, the telemetry.* namespace uses diskless topics because scrape intervals are 15 seconds or longer - the added latency is invisible against that interval. The cognition.* namespace uses standard local-disk topics because cognitive loop latency is sensitive to pipeline delays.