Question on Flink and Rest API

[Ejaskhan S]

Team,

It's just a random thought. 

Can I make the Flink application exposing a rest endpoint for the data source? So a client could send data to this endpoint. Subsequently, Flink processed this data and responded to the client application through the endpoint, like a client-server model.

Thanks 

EK

[raghav280392]

Hi Ejaskhan

As per my understanding, this approach will require your data source to run a HTTP server within itself (embedded web server) and I am not sure If it is a good design. It looks like you are trying to build a synchronous(client-server model) processing model in Flink. But Flink is meant for processing data asynchronously (streaming/batch). Can you elaborate on the use case which you are trying to address? What kind of processing are you planning to do in Flink?

Check If the following approach makes sense for your use case.

Accept the events from HTTP client and persist the events to a data store (SQL/NoSQL) and publish the same to Kafka(topic)/RabbitMQ(queue). Let Flink data source listen to these topics/queues and update the status (SUCCESS/FAILURE) in the data store. If your clients are ok to see some lag in events, you can directly publish the events to Kafka(topic)/RabbitMQ(queue) without persisting it in the data store. Let Flink do all the processing and finally write to the data store.

Thank you

Raghavendar T S

https://www.linkedin.com/in/raghavendar-ts

On Wed, Feb 3, 2021 at 11:29 AM Ejaskhan S <[hidden email]> wrote:

Team,

It's just a random thought. 

Can I make the Flink application exposing a rest endpoint for the data source? So a client could send data to this endpoint. Subsequently, Flink processed this data and responded to the client application through the endpoint, like a client-server model.

Thanks 

EK

--

[Tzu-Li (Gordon) Tai]

Hi,

There is no out-of-box Flink source/sink connector for this, but it isn't
unheard of that users have implemented something to support what you
outlined.

One way to possibly achieve this is: in terms of a Flink streaming job
graph, what you would need to do is co-locate the source (which exposes the
endpoint and maintains a pool of open client connections mapped by request
ID), and the sink operators (which receives processed results with the
original request IDs attached, and is in charge for replying to the original
requests). The open client connections need to be process-wide accessible
(e.g. via a static reference), so that when a co-located sink operator
receives a result, it can directly fetch the corresponding client connection
and return a response.

The specifics are of course a bit more evolved; probably need some digging
around previous Flink Forward conference talks to get a better picture.
Hopefully this gives you a starting point to think about.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Ejaskhan S]

Hi Raghavendar,

Yes , you are right. Your approach is correct ,and it is the most straightforward one.but I was just thinking about the possibilities of my question mentioned.

Thanks

EK

On Wed, Feb 3, 2021, 12:02 PM Raghavendar T S <[hidden email]> wrote:

Hi Ejaskhan

As per my understanding, this approach will require your data source to run a HTTP server within itself (embedded web server) and I am not sure If it is a good design. It looks like you are trying to build a synchronous(client-server model) processing model in Flink. But Flink is meant for processing data asynchronously (streaming/batch). Can you elaborate on the use case which you are trying to address? What kind of processing are you planning to do in Flink?

Check If the following approach makes sense for your use case.

Accept the events from HTTP client and persist the events to a data store (SQL/NoSQL) and publish the same to Kafka(topic)/RabbitMQ(queue). Let Flink data source listen to these topics/queues and update the status (SUCCESS/FAILURE) in the data store. If your clients are ok to see some lag in events, you can directly publish the events to Kafka(topic)/RabbitMQ(queue) without persisting it in the data store. Let Flink do all the processing and finally write to the data store.

Thank you

Raghavendar T S

https://www.linkedin.com/in/raghavendar-ts

On Wed, Feb 3, 2021 at 11:29 AM Ejaskhan S <[hidden email]> wrote:

Team,

It's just a random thought. 

Can I make the Flink application exposing a rest endpoint for the data source? So a client could send data to this endpoint. Subsequently, Flink processed this data and responded to the client application through the endpoint, like a client-server model.

Thanks 

EK

--

Raghavendar T S

www.teknosrc.com

[Ejaskhan S]

Yes Gordon, it's obviously gave me a starting point to think about.

On Wed, Feb 3, 2021, 12:02 PM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi,

There is no out-of-box Flink source/sink connector for this, but it isn't
unheard of that users have implemented something to support what you
outlined.

One way to possibly achieve this is: in terms of a Flink streaming job
graph, what you would need to do is co-locate the source (which exposes the
endpoint and maintains a pool of open client connections mapped by request
ID), and the sink operators (which receives processed results with the
original request IDs attached, and is in charge for replying to the original
requests). The open client connections need to be process-wide accessible
(e.g. via a static reference), so that when a co-located sink operator
receives a result, it can directly fetch the corresponding client connection
and return a response.

The specifics are of course a bit more evolved; probably need some digging
around previous Flink Forward conference talks to get a better picture.
Hopefully this gives you a starting point to think about.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/