Consumer group
A named group of Kafka consumers that jointly consume a topic, with each partition assigned to exactly one consumer in the group at a time. Consumer groups enable parallel processing and horizontal scaling of pipeline workers. Group lag (the difference between the latest offset and the committed offset) is the primary autoscaling signal for KEDA.
Kafka's consumer group mechanism is what makes horizontal scaling of cognitive workers possible. Each topic partition is consumed by exactly one member of a consumer group at any moment, ensuring ordering within a partition while allowing multiple partitions to be processed in parallel. Adding a consumer to the group triggers rebalancing: partitions are redistributed so the new consumer takes on its share of the load. The key operational metric for a consumer group is lag - how many messages are waiting to be processed. Rising lag signals that the current group size cannot keep up with the producer rate, which is the trigger for KEDA to add worker replicas. Consumer group committed offsets also enable replay: by resetting offsets, operators can re-process historical events for debugging or re-indexing.