Change sink topology

Posted by Smirnov Sergey Vladimirovich (39833) on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Change-sink-topology-tp28143.html

Hi flink,

 

Im wonder, is it possible to dynamically (while job running) change sink topology* - by adding new sink on the fly?

Say, we have input stream and by analyzing message property we decided to put this message into some kafka topic, i.e. choosen_topic = function(message.property).

Simplifying: sink_topic = ‘logger_group_’+message.groupId

When job was launched we don`t know list of all possible groupId.

These sink topics (and groupId) are created (and even removed) dynamically upon executing some external rules during the entire job period.

 

 

Best, Sergey