Couldn't figure out - How to do this in Flink? - Pls assist with suggestions

[Titus Rakkesh]

Dears,

I have a data stream continuously coming,

DataStream<Tuple3<String, Integer, Double>> splitZTuple;

Eg  - (775168263,113182,0.0)

I have to store this for 24 hrs expiry in somewhere (Window or somewhere) to check against another stream.

The second stream is

DataStream<Tuple2<String, Double>> splittedVomsTuple which also continuously receiving one.

Eg. (775168263,100.0)

We need to accumulate the third element in (775168263,113182,0.0) in the WINDOW (If the corresponding first element match happened with the incoming second streams second element 775168263,100.0)

While keeping this WINDOW session if any (775168263,113182,175) third element in the Window Stream exceed a value (Eg >150) we need to call back a another rest point to send an alert ------- (775168263,113182,175) match the criteria. Simply a CEP call back.

In Flink how we can do this kind of operations? Or do I need to think about any other framework? Please advise.

Thanks...

[Chesnay Schepler]

You should be able to use a KeyedProcessFunction for that.
Find matching elements via keyBy() on the first field.
Aggregate into ValueState, send alert if necessary.
Upon encountering a new key, setup a timer to remove the entry in 24h.

On 08.02.2019 07:43, Titus Rakkesh wrote:

Dears,

I have a data stream continuously coming,

DataStream<Tuple3<String, Integer, Double>> splitZTuple;

Eg  - (775168263,113182,0.0)

I have to store this for 24 hrs expiry in somewhere (Window or somewhere) to check against another stream.

The second stream is

DataStream<Tuple2<String, Double>> splittedVomsTuple which also continuously receiving one.

Eg. (775168263,100.0)

We need to accumulate the third element in (775168263,113182,0.0) in the WINDOW (If the corresponding first element match happened with the incoming second streams second element 775168263,100.0)

While keeping this WINDOW session if any (775168263,113182,175) third element in the Window Stream exceed a value (Eg >150) we need to call back a another rest point to send an alert ------- (775168263,113182,175) match the criteria. Simply a CEP call back.

In Flink how we can do this kind of operations? Or do I need to think about any other framework? Please advise.

Thanks...

[Titus Rakkesh]

Thanks Chesnay. I will try that and let you know.

Thanks.

On Sun, Feb 10, 2019 at 2:31 PM Chesnay Schepler <[hidden email]> wrote:

You should be able to use a KeyedProcessFunction for that.
Find matching elements via keyBy() on the first field.
Aggregate into ValueState, send alert if necessary.
Upon encountering a new key, setup a timer to remove the entry in 24h.

On 08.02.2019 07:43, Titus Rakkesh wrote:

Dears,

I have a data stream continuously coming,

DataStream<Tuple3<String, Integer, Double>> splitZTuple;

Eg  - (775168263,113182,0.0)

I have to store this for 24 hrs expiry in somewhere (Window or somewhere) to check against another stream.

The second stream is

DataStream<Tuple2<String, Double>> splittedVomsTuple which also continuously receiving one.

Eg. (775168263,100.0)

We need to accumulate the third element in (775168263,113182,0.0) in the WINDOW (If the corresponding first element match happened with the incoming second streams second element 775168263,100.0)

While keeping this WINDOW session if any (775168263,113182,175) third element in the Window Stream exceed a value (Eg >150) we need to call back a another rest point to send an alert ------- (775168263,113182,175) match the criteria. Simply a CEP call back.

In Flink how we can do this kind of operations? Or do I need to think about any other framework? Please advise.

Thanks...