How to add new Kafka topic to consumer

[chandresh pancholi]

Hi,

Is there a way to add Kafka topic dynamically to stream?

For example there were Kafka topic named foo1, foo2, foo3 and task manager will start consuming events from all 3 topics. Now I create another two topic foo4 & bar1 in Kafka.

How will FlinkKafkaConsumer would read events from foo4 & bar1 topic?

Thanks

--

[Alexander Smirnov]

this feature has been implemented in 1.4.0, take a look at 

https://issues.apache.org/jira/browse/FLINK-4022

https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/connectors/kafka.html#kafka-consumers-topic-and-partition-discovery

On Wed, Apr 11, 2018 at 3:33 PM chandresh pancholi <[hidden email]> wrote:

Hi,

Is there a way to add Kafka topic dynamically to stream?

For example there were Kafka topic named foo1, foo2, foo3 and task manager will start consuming events from all 3 topics. Now I create another two topic foo4 & bar1 in Kafka.

How will FlinkKafkaConsumer would read events from foo4 & bar1 topic?

Thanks

--

[Chesnay Schepler]

In 1.3 and below I believe this is not possible.

For 1.4 and above, please see https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/kafka.html#partition-discovery.

On 11.04.2018 14:33, chandresh pancholi wrote:

Hi,

Is there a way to add Kafka topic dynamically to stream?

For example there were Kafka topic named foo1, foo2, foo3 and task manager will start consuming events from all 3 topics. Now I create another two topic foo4 & bar1 in Kafka.

How will FlinkKafkaConsumer would read events from foo4 & bar1 topic?

Thanks

--