Partition
A subdivision of a Kafka topic that enables parallel processing. Each partition is an ordered, independent log consumed by at most one consumer in a consumer group at a time. Partition count determines the maximum achievable parallelism for a topic's consumer group.
Partition count is the fundamental scaling parameter for Kafka throughput. A topic with N partitions can be consumed by at most N parallel consumers in a single consumer group, and events within a partition are delivered in order. The partitioning key determines which partition an event is routed to: using a consistent key (such as agent ID or session ID) ensures that events for the same logical entity are always handled by the same consumer, preserving ordering for dependent events. Choosing partition count is a capacity planning decision: too few partitions create a throughput ceiling that cannot be raised without rebalancing; too many partitions increase overhead and coordination cost. In the Cognitive Substrate, high-volume telemetry topics use more partitions than lower-volume cognition topics, reflecting their different throughput requirements.