Horizontal scaling
A scaling property in which agent workers and pipeline components scale by adding parallel replicas rather than larger machines, coordinated via Kafka partitions and consumer groups.
The Cognitive Substrate is designed from the ground up for horizontal scaling: every pipeline component that does meaningful work can scale by adding replicas rather than requiring a larger machine. This works because the cognitive bus decouples producers from consumers via Kafka topics with explicit partitioning. Enrichment workers, consolidation workers, reinforcement workers, pattern detection workers, and agent workers are all stateless consumers on their respective topics - adding a replica means adding another consumer to the group, which automatically takes on a share of the partitions via Kafka's rebalancing protocol. The practical limit is partition count: a topic with 12 partitions can be consumed by at most 12 parallel workers in a single consumer group. KEDA automates this scaling loop: it reads consumer lag from Kafka, computes the appropriate replica count, and adjusts the deployment without manual intervention. The only components that do not scale horizontally in the same way are the ones with strong ordering requirements - the arbitration layer within a session must remain sequential within that session to preserve decision coherence.