Re: Change sink topology

Posted by Fabian Hueske-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Change-sink-topology-tp28143p28144.html

Hi Sergey,

I would not consider this to be a topology change (the sink operator would still be a Kafka producer).

It seems that dynamic topic selection is possible with a KeyedSerializationSchema (Check "Advanced Serialization Schema: [1]).

Best,

Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-producer

Am Do., 6. Juni 2019 um 13:50 Uhr schrieb Smirnov Sergey Vladimirovich <[hidden email]>:

Hi flink,

 

Im wonder, is it possible to dynamically (while job running) change sink topology* - by adding new sink on the fly?

Say, we have input stream and by analyzing message property we decided to put this message into some kafka topic, i.e. choosen_topic = function(message.property).

Simplifying: sink_topic = ‘logger_group_’+message.groupId

When job was launched we don`t know list of all possible groupId.

These sink topics (and groupId) are created (and even removed) dynamically upon executing some external rules during the entire job period.

 

 

Best, Sergey