Broker-visible vs client-local: rethinking parallelism in Kafka

Kafka share groups were built for queue semantics, not just parallel consumption. The author argues that broker-visible parallelism—scaling consumers and partitions—creates massive overhead in connections and state for high-throughput workloads. Client-local parallelism via virtual threads or async tasks achieves the same throughput with far fewer consumers and TCP connections, though it adds client complexity. The post advocates for hybrid approaches and reviving client-side parallel processing libraries for share groups.