Flink & Kafka multi-node config

[Michael Latta]

If I use defaults for the most part but configure flink to have parallelism 5 and kafka to have 5 brokers (one of each on 5 nodes) will the connector and kafka be smart enough to use the kafka partition on the same node as the flink task manager for the 5 partitions?  Do I need to explicitly assign partitions to get them local to each other?

Michael

[Tzu-Li (Gordon) Tai]

Hi,

The partition-to-subtask assignment of is not locality aware.

There were discussions to expose functionality for custom user-defined
assignment methods, which it might be possible to leverage that for a
locality aware assignment.
Unfortunately, this feature is not implemented, yet.

The rrelated issue that tracks this feature is
https://issues.apache.org/jira/browse/FLINK-8570.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/