Re: About Kafka08Fetcher and Kafka010Fetcher

Posted by Tzu-Li (Gordon) Tai on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/About-Kafka08Fetcher-and-Kafka010Fetcher-tp17499p17620.html

Hi Jaxon,

The threading model is implemented differently between the Kafka08Fetcher and all other fetcher versions higher than 0.9+ because the Kafka Java clients used between these versions have different abstraction levels.

The Kafka08Fetcher still uses the low-level `SimpleConsumer` API, which only allows connection to a single Kafka broker. Therefore, the Kafka08Fetcher maintains a thread for each broker that it connects to. If a subtask’s assigned partitions all exist on the same Kafka broker, only one thread will be created.

On the other hand, in versions 0.9+, the high-level `KafkaConsumer` API is used, which hides away the complexity for per-broker connections. I believe that the `KafkaConsumer` client still creates per-broker threads internally, if its assigned partitions span multiple brokers.

Cheers,

Gordon

On 2 January 2018 at 8:22:22 AM, Timo Walther ([hidden email]) wrote:

Maybe Gordon (in CC) can answer your question.


Am 1/1/18 um 3:36 PM schrieb Jaxon Hu:
> In Kafka08Fetcher, it use  Map<Node,SimpleConsumerThread> to manage
> multi-threads. But I notice in Kafka09Fetcher or Kafka010Fetcher, it's
> gone. So how Kafka09Fetcher implements multi-threads read partitions
> from kafka?