Broker-Visible vs Client-Local Parallelism — Jack Vanlightly
This post is a little side-quest from my “Kafka Share Groups and Parallelizing Consumption” series. My “Kafka Share Groups and Parallelizing Consumption” series ( part 1 , part 2 ) has been laser focused on how different configurations and behaviors affect parallel consumption in share groups. So far I’ve shown that you most definitely can hold share groups wrong . You could quite easily and inadvertently create a work queue and with the right combination of things going against you, see a small number of consumers dominate, leaving most consumers starved of messages. All the while lag builds and builds. You need to know the settings and what they do. But it’s worth asking the question: is parallelizing consumption what share groups are for?
Jack Vanlightly