Hi Jaxon,
The threading model is implemented differently between the Kafka08Fetcher and all other fetcher versions higher than 0.9+ because the Kafka Java clients used between these versions have different abstraction levels.
The Kafka08Fetcher still uses the low-level `SimpleConsumer` API, which only allows connection to a single Kafka broker. Therefore, the Kafka08Fetcher maintains a thread for each broker that it connects to. If a subtask’s assigned partitions all exist on the same Kafka broker, only one thread will be created.
On the other hand, in versions 0.9+, the high-level `KafkaConsumer` API is used, which hides away the complexity for per-broker connections. I believe that the `KafkaConsumer` client still creates per-broker threads internally, if its assigned partitions span multiple brokers.
Cheers,
Gordon
On 2 January 2018 at 8:22:22 AM, Timo Walther ([hidden email]) wrote:
Maybe Gordon (in CC) can answer your question.
Am 1/1/18 um 3:36 PM schrieb Jaxon Hu:
> In Kafka08Fetcher, it use Map<Node,SimpleConsumerThread> to manage
> multi-threads. But I notice in Kafka09Fetcher or Kafka010Fetcher, it's
> gone. So how Kafka09Fetcher implements multi-threads read partitions
> from kafka?
Locked
