Routing events to different kafka topics dynamically

[Prasanna kumar]

Hi,

Events need to be routed to different kafka topics dynamically based upon some info in the message.

We have implemented using KeyedSerializationSchema similar to https://stackoverflow.com/questions/49508508/apache-flink-how-to-sink-events-to-different-kafka-topics-depending-on-the-even. But its deprecated and we cannot use it for production. 

I looked at the alternative KafkaSerializationSchema but there i do not see an option there.

Then i stumbled upon this http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Usage-of-KafkaDeserializationSchema-and-KafkaSerializationSchema-td32347.html. which asks us to use KafkaContextAware.

Is there a more intuitive/easier way to do the same ? 

Thanks,

Prasanna.

[Till Rohrmann]

Hi Prasanna,

I believe that what Aljoscha suggestd in the linked discussion is still the best way to go forward. Given your description of the problem this should actually be pretty straightforward as you can deduce the topic from the message. Hence, you just need to create the ProducerRecord with the right target topic you extracted from the record/message.

Cheers,

Till

On Wed, Dec 2, 2020 at 5:28 PM Prasanna kumar <[hidden email]> wrote:

Hi,

Events need to be routed to different kafka topics dynamically based upon some info in the message.

We have implemented using KeyedSerializationSchema similar to https://stackoverflow.com/questions/49508508/apache-flink-how-to-sink-events-to-different-kafka-topics-depending-on-the-even. But its deprecated and we cannot use it for production. 

I looked at the alternative KafkaSerializationSchema but there i do not see an option there.

Then i stumbled upon this http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Usage-of-KafkaDeserializationSchema-and-KafkaSerializationSchema-td32347.html. which asks us to use KafkaContextAware.

Is there a more intuitive/easier way to do the same ? 

Thanks,

Prasanna.

[Prasanna kumar]

Thanks Till,

Able to deduce topics by extending the KafkaSerializarion Schema class. 

Prasanna.

On Wed, Dec 2, 2020 at 11:18 PM Till Rohrmann <[hidden email]> wrote:

Hi Prasanna,

I believe that what Aljoscha suggestd in the linked discussion is still the best way to go forward. Given your description of the problem this should actually be pretty straightforward as you can deduce the topic from the message. Hence, you just need to create the ProducerRecord with the right target topic you extracted from the record/message.

Cheers,

Till

On Wed, Dec 2, 2020 at 5:28 PM Prasanna kumar <[hidden email]> wrote:

Hi,

Events need to be routed to different kafka topics dynamically based upon some info in the message.

We have implemented using KeyedSerializationSchema similar to https://stackoverflow.com/questions/49508508/apache-flink-how-to-sink-events-to-different-kafka-topics-depending-on-the-even. But its deprecated and we cannot use it for production. 

I looked at the alternative KafkaSerializationSchema but there i do not see an option there.

Then i stumbled upon this http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Usage-of-KafkaDeserializationSchema-and-KafkaSerializationSchema-td32347.html. which asks us to use KafkaContextAware.

Is there a more intuitive/easier way to do the same ? 

Thanks,

Prasanna.

[Till Rohrmann]

Great to hear :-)

Cheers,

Till

On Thu, Dec 3, 2020 at 10:15 AM Prasanna kumar <[hidden email]> wrote:

Thanks Till,

Able to deduce topics by extending the KafkaSerializarion Schema class. 

Prasanna.

On Wed, Dec 2, 2020 at 11:18 PM Till Rohrmann <[hidden email]> wrote:

Hi Prasanna,

I believe that what Aljoscha suggestd in the linked discussion is still the best way to go forward. Given your description of the problem this should actually be pretty straightforward as you can deduce the topic from the message. Hence, you just need to create the ProducerRecord with the right target topic you extracted from the record/message.

Cheers,

Till

On Wed, Dec 2, 2020 at 5:28 PM Prasanna kumar <[hidden email]> wrote:

Hi,

Events need to be routed to different kafka topics dynamically based upon some info in the message.

We have implemented using KeyedSerializationSchema similar to https://stackoverflow.com/questions/49508508/apache-flink-how-to-sink-events-to-different-kafka-topics-depending-on-the-even. But its deprecated and we cannot use it for production. 

I looked at the alternative KafkaSerializationSchema but there i do not see an option there.

Then i stumbled upon this http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Usage-of-KafkaDeserializationSchema-and-KafkaSerializationSchema-td32347.html. which asks us to use KafkaContextAware.

Is there a more intuitive/easier way to do the same ? 

Thanks,

Prasanna.