Is Apache Flink suitable for an application where messages are routed to different services

[Thamidu Muthukumarana]

Hello,

I'm exploring different options for my use case, which I will describe in detail below. Few of the options considered include apache kafka and flink.

Use Case Description

When a message comes to the application, it needs to extract the configurations set by the user who authored the message.

The configuration will contain third party services (e.g. : like Facebook, Twitter) and access tokens(of the user) for these third parties

Then it needs to forward these messages to the specified third parties using the stored access tokens.

Basically, the application should support forwarding messages to third parties using access tokens stored by the user.

I want to know if Apache flink is suitable for this purpose. I feel that it can be used but wanted to know what people who are more familiar with flink think. Thank you.

Regards,

Thamidu

[Ardhani Narasimha Swamy]

Hi Thamidu,

I have been working on flink for the past few months, below is some analysis based on your use case. 

Few questions

1. What is the volume of data in the context? 

2. What is the TPS that you are looking for?

3. Is it fine if the same message gets forwarded to 3rd party service multiple times?

4. Is there any inter-dependency. b/w the events coming for the same user?

5. Do you need to retain state for any purpose?

Your use case seems to be very simple at a very high level, where you would want to dispatch requests based on the message received.

Flow if I understand correctly is

KAFKA --> Message Dispatcher --> ThirdPart Services. 

If you want a simple stateless message dispatcher, then you can build small lightweight apps (node, go) to dispatch them respectively. 

Based on the traffic nodes can be auto scaled up/down (assuming you're gonna deploy on cloud, specifically k8s), which could incur lower infra cost IMO.

Flink will as well do the job, but it has its own merits for appropriate use cases. 

These are all my views, let's wait for what experts have to say. 

Thanks,

Narasimha

On Tue, Nov 3, 2020 at 6:38 PM Thamidu Muthukumarana <[hidden email]> wrote:

Hello,

I'm exploring different options for my use case, which I will describe in detail below. Few of the options considered include apache kafka and flink.

Use Case Description

When a message comes to the application, it needs to extract the configurations set by the user who authored the message.

The configuration will contain third party services (e.g. : like Facebook, Twitter) and access tokens(of the user) for these third parties

Then it needs to forward these messages to the specified third parties using the stored access tokens.

Basically, the application should support forwarding messages to third parties using access tokens stored by the user.

I want to know if Apache flink is suitable for this purpose. I feel that it can be used but wanted to know what people who are more familiar with flink think. Thank you.

Regards,

Thamidu

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

IMPORTANT: The contents of this email and any attachments are confidential and protected by applicable laws. If you have received this email by mistake, please (i) notify the sender immediately; (ii) delete it from your database; and (iii) do not disclose the contents to anyone or make copies thereof. Razorpay accepts no liability caused due to any inadvertent/ unintentional data transmitted through this email.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

[raghav280392]

Hi Thamidu

You can very well fit your use case in Flink. Also your use case looks simple based on your post. You can also check Kafka Streams which can run as standalone applications. If your use case is simple, Flink is not necessary.

Thank you

On Tue, Nov 3, 2020 at 7:42 PM Ardhani Narasimha Swamy <[hidden email]> wrote:

Hi Thamidu,

I have been working on flink for the past few months, below is some analysis based on your use case. 

Few questions

1. What is the volume of data in the context? 

2. What is the TPS that you are looking for?

3. Is it fine if the same message gets forwarded to 3rd party service multiple times?

4. Is there any inter-dependency. b/w the events coming for the same user?

5. Do you need to retain state for any purpose?

Your use case seems to be very simple at a very high level, where you would want to dispatch requests based on the message received.

Flow if I understand correctly is

KAFKA --> Message Dispatcher --> ThirdPart Services. 

If you want a simple stateless message dispatcher, then you can build small lightweight apps (node, go) to dispatch them respectively. 

Based on the traffic nodes can be auto scaled up/down (assuming you're gonna deploy on cloud, specifically k8s), which could incur lower infra cost IMO.

Flink will as well do the job, but it has its own merits for appropriate use cases. 

These are all my views, let's wait for what experts have to say. 

Thanks,

Narasimha

On Tue, Nov 3, 2020 at 6:38 PM Thamidu Muthukumarana <[hidden email]> wrote:

Hello,

I'm exploring different options for my use case, which I will describe in detail below. Few of the options considered include apache kafka and flink.

Use Case Description

When a message comes to the application, it needs to extract the configurations set by the user who authored the message.

The configuration will contain third party services (e.g. : like Facebook, Twitter) and access tokens(of the user) for these third parties

Then it needs to forward these messages to the specified third parties using the stored access tokens.

Basically, the application should support forwarding messages to third parties using access tokens stored by the user.

I want to know if Apache flink is suitable for this purpose. I feel that it can be used but wanted to know what people who are more familiar with flink think. Thank you.

Regards,

Thamidu

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

IMPORTANT: The contents of this email and any attachments are confidential and protected by applicable laws. If you have received this email by mistake, please (i) notify the sender immediately; (ii) delete it from your database; and (iii) do not disclose the contents to anyone or make copies thereof. Razorpay accepts no liability caused due to any inadvertent/ unintentional data transmitted through this email.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

--

Raghavendar T S

www.teknosrc.com