Change sink topology

[Smirnov Sergey Vladimirovich (39833)]

Hi flink,

 

Im wonder, is it possible to dynamically (while job running) change sink topology* - by adding new sink on the fly?

Say, we have input stream and by analyzing message property we decided to put this message into some kafka topic, i.e. choosen_topic = function(message.property).

Simplifying: sink_topic = ‘logger_group_’+message.groupId

When job was launched we don`t know list of all possible groupId.

These sink topics (and groupId) are created (and even removed) dynamically upon executing some external rules during the entire job period.

 

 

Best, Sergey

[Fabian Hueske-2]

Hi Sergey,

I would not consider this to be a topology change (the sink operator would still be a Kafka producer).

It seems that dynamic topic selection is possible with a KeyedSerializationSchema (Check "Advanced Serialization Schema: [1]).

Best,

Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-producer

Am Do., 6. Juni 2019 um 13:50 Uhr schrieb Smirnov Sergey Vladimirovich <[hidden email]>:

Hi flink,

 

Im wonder, is it possible to dynamically (while job running) change sink topology* - by adding new sink on the fly?

Say, we have input stream and by analyzing message property we decided to put this message into some kafka topic, i.e. choosen_topic = function(message.property).

Simplifying: sink_topic = ‘logger_group_’+message.groupId

When job was launched we don`t know list of all possible groupId.

These sink topics (and groupId) are created (and even removed) dynamically upon executing some external rules during the entire job period.

 

 

Best, Sergey

[Smirnov Sergey Vladimirovich (39833)]

Great, thanks!

 

From: Fabian Hueske [mailto:[hidden email]]
Sent: Thursday, June 6, 2019 3:07 PM
To: Smirnov Sergey Vladimirovich <[hidden email]>
Cc: [hidden email]
Subject: Re: Change sink topology

 

Hi Sergey,

 

I would not consider this to be a topology change (the sink operator would still be a Kafka producer).

It seems that dynamic topic selection is possible with a KeyedSerializationSchema (Check "Advanced Serialization Schema: [1]).

 

Best,

Fabian

 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-producer

 

Am Do., 6. Juni 2019 um 13:50 Uhr schrieb Smirnov Sergey Vladimirovich <[hidden email]>:

Hi flink,

 

Im wonder, is it possible to dynamically (while job running) change sink topology* - by adding new sink on the fly?

Say, we have input stream and by analyzing message property we decided to put this message into some kafka topic, i.e. choosen_topic = function(message.property).

Simplifying: sink_topic = ‘logger_group_’+message.groupId

When job was launched we don`t know list of all possible groupId.

These sink topics (and groupId) are created (and even removed) dynamically upon executing some external rules during the entire job period.

 

 

Best, Sergey