KEDA

KEDA (Kubernetes Event-Driven Autoscaler) bridges the gap between Kafka's consumer lag metrics and Kubernetes deployment scaling. Standard Kubernetes Horizontal Pod Autoscaler scales on CPU and memory, which are poor proxies for Kafka processing need: a worker can be CPU-idle but have 100,000 messages queued because the bottleneck is I/O or downstream latency. KEDA reads the lag directly from Kafka consumer groups and translates it into a replica count: if lag exceeds the configured threshold per replica, KEDA adds replicas until lag falls back within bounds or the maximum replica count is reached. When lag returns to zero, KEDA scales back down to the configured minimum. In the Cognitive Substrate, KEDA manages the enrichment workers, consolidation workers, reinforcement workers, and pattern detection workers - all the stateless pipeline workers where throughput scales linearly with replica count up to the partition limit. The agent workers within active sessions are managed differently, as they have session-scoped state that cannot be freely redistributed across replicas mid-session.