Flink event processing immediate feedback

[igor.berman]

Hi,
Suppose I have web facing frontend that gets stream of events(http calls). I need to process event stream and do some aggregations over those events and write aggregated statistics to Hbase - so far Flink seems as perfect match.
However in some cases event should trigger some alert and frontend needs to get this alert in synchronous way - here I'm a bit lost. I thought about some kind of following flow:
frontend -> queue -> flink -> redis(pub/sub)<- frontend

I.e. I have two major use cases - async aggregated analytics/stats computing and "synchronous" response to frontend. Frontend might be node/play or any other technology that won't have a problem of "waiting" for the response, so the only question - how to implement this feedback ?
Might be some kind of Sink?

Any ideas would be appreciated,
Igor

[Aljoscha Krettek]

Hi,

what do you mean by "synchronous". If I understood it correctly then some events entering the Flink pipeline would trigger an alert while some others would not trigger an alert. How would the component that receives such alerts know when to wait and when to don't wait.

 As I see it you can push these alerts into some sort of message queue and have a request in the front-end that returns as soon as something is available in that queue. Then you display it and start a new query that returns as soon as more data is available in the alert queue.

Cheers,

Aljoscha

On Thu, 7 Apr 2016 at 09:57 igor.berman <[hidden email]> wrote:

Hi,
Suppose I have web facing frontend that gets stream of events(http calls). I
need to process event stream and do some aggregations over those events and
write aggregated statistics to Hbase - so far Flink seems as perfect match.
However in some cases event should trigger some alert and frontend needs to
get this alert in synchronous way - here I'm a bit lost. I thought about
some kind of following flow:
frontend -> queue -> flink -> redis(pub/sub)<- frontend

I.e. I have two major use cases - async aggregated analytics/stats computing
and "synchronous" response to frontend. Frontend might be node/play or any
other technology that won't have a problem of "waiting" for the response, so
the only question - how to implement this feedback ?
Might be some kind of Sink?

Any ideas would be appreciated,
Igor




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-event-processing-immediate-feedback-tp5978.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[igor.berman]

Hi Aljoscha,

thanks for the response

Synchronous - in our case means that request by end-client to frontend(say some REST call) needs to wait until processing in backend(Flink) is done and should return response(e.g. alert) back to end-client(i.e. end-client -> frontend -> kafka-> flink)

those request are minority and will be defined by different type of request to frontend or different parameters

Might be Flink is not best option for this use case?

Igor

 

On 14 April 2016 at 16:37, Aljoscha Krettek <[hidden email]> wrote:

Hi,

what do you mean by "synchronous". If I understood it correctly then some events entering the Flink pipeline would trigger an alert while some others would not trigger an alert. How would the component that receives such alerts know when to wait and when to don't wait.

 As I see it you can push these alerts into some sort of message queue and have a request in the front-end that returns as soon as something is available in that queue. Then you display it and start a new query that returns as soon as more data is available in the alert queue.

Cheers,

Aljoscha

On Thu, 7 Apr 2016 at 09:57 igor.berman <[hidden email]> wrote:

Hi,
Suppose I have web facing frontend that gets stream of events(http calls). I
need to process event stream and do some aggregations over those events and
write aggregated statistics to Hbase - so far Flink seems as perfect match.
However in some cases event should trigger some alert and frontend needs to
get this alert in synchronous way - here I'm a bit lost. I thought about
some kind of following flow:
frontend -> queue -> flink -> redis(pub/sub)<- frontend

I.e. I have two major use cases - async aggregated analytics/stats computing
and "synchronous" response to frontend. Frontend might be node/play or any
other technology that won't have a problem of "waiting" for the response, so
the only question - how to implement this feedback ?
Might be some kind of Sink?

Any ideas would be appreciated,
Igor




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-event-processing-immediate-feedback-tp5978.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Aljoscha Krettek]

Could still be, as I described it by using a message queue to do the communication between Flink and the front end.

On Thu, 14 Apr 2016 at 17:30 Igor Berman <[hidden email]> wrote:

Hi Aljoscha,

thanks for the response

Synchronous - in our case means that request by end-client to frontend(say some REST call) needs to wait until processing in backend(Flink) is done and should return response(e.g. alert) back to end-client(i.e. end-client -> frontend -> kafka-> flink)

those request are minority and will be defined by different type of request to frontend or different parameters

Might be Flink is not best option for this use case?

Igor

 

On 14 April 2016 at 16:37, Aljoscha Krettek <[hidden email]> wrote:

Hi,

what do you mean by "synchronous". If I understood it correctly then some events entering the Flink pipeline would trigger an alert while some others would not trigger an alert. How would the component that receives such alerts know when to wait and when to don't wait.

 As I see it you can push these alerts into some sort of message queue and have a request in the front-end that returns as soon as something is available in that queue. Then you display it and start a new query that returns as soon as more data is available in the alert queue.

Cheers,

Aljoscha

On Thu, 7 Apr 2016 at 09:57 igor.berman <[hidden email]> wrote:

Hi,
Suppose I have web facing frontend that gets stream of events(http calls). I
need to process event stream and do some aggregations over those events and
write aggregated statistics to Hbase - so far Flink seems as perfect match.
However in some cases event should trigger some alert and frontend needs to
get this alert in synchronous way - here I'm a bit lost. I thought about
some kind of following flow:
frontend -> queue -> flink -> redis(pub/sub)<- frontend

I.e. I have two major use cases - async aggregated analytics/stats computing
and "synchronous" response to frontend. Frontend might be node/play or any
other technology that won't have a problem of "waiting" for the response, so
the only question - how to implement this feedback ?
Might be some kind of Sink?

Any ideas would be appreciated,
Igor




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-event-processing-immediate-feedback-tp5978.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[igor.berman]

Yes, indeed this is direction we are trying currently

thanks

On 14 April 2016 at 18:31, Aljoscha Krettek <[hidden email]> wrote:

Could still be, as I described it by using a message queue to do the communication between Flink and the front end.

On Thu, 14 Apr 2016 at 17:30 Igor Berman <[hidden email]> wrote:

Hi Aljoscha,

thanks for the response

Synchronous - in our case means that request by end-client to frontend(say some REST call) needs to wait until processing in backend(Flink) is done and should return response(e.g. alert) back to end-client(i.e. end-client -> frontend -> kafka-> flink)

those request are minority and will be defined by different type of request to frontend or different parameters

Might be Flink is not best option for this use case?

Igor

 

On 14 April 2016 at 16:37, Aljoscha Krettek <[hidden email]> wrote:

Hi,

what do you mean by "synchronous". If I understood it correctly then some events entering the Flink pipeline would trigger an alert while some others would not trigger an alert. How would the component that receives such alerts know when to wait and when to don't wait.

 As I see it you can push these alerts into some sort of message queue and have a request in the front-end that returns as soon as something is available in that queue. Then you display it and start a new query that returns as soon as more data is available in the alert queue.

Cheers,

Aljoscha

On Thu, 7 Apr 2016 at 09:57 igor.berman <[hidden email]> wrote:

Hi,
Suppose I have web facing frontend that gets stream of events(http calls). I
need to process event stream and do some aggregations over those events and
write aggregated statistics to Hbase - so far Flink seems as perfect match.
However in some cases event should trigger some alert and frontend needs to
get this alert in synchronous way - here I'm a bit lost. I thought about
some kind of following flow:
frontend -> queue -> flink -> redis(pub/sub)<- frontend

I.e. I have two major use cases - async aggregated analytics/stats computing
and "synchronous" response to frontend. Frontend might be node/play or any
other technology that won't have a problem of "waiting" for the response, so
the only question - how to implement this feedback ?
Might be some kind of Sink?

Any ideas would be appreciated,
Igor




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-event-processing-immediate-feedback-tp5978.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.