Regarding time window based on the values received in the stream

[Abdul Salam Shaikh]

Hi,

I have a requirement for my thesis project where I need to set the time window based on the value which I received in the event stream. 

I need to collect the values when the value starts at 0 and increments to a max value(which is unknown) till the value resets to 0 again on which a trigger is triggered which performs a calculation based on the values collected. 

DataStream<Tuple4<String,Double,String,Integer>> detectorEventStream = streamMatchedTuples

        .keyBy(Based on the first param of the tuple)

        .timeWindow(based on the second param of the tuple)

        .apply(new PhaseCalculator());

I am relatively new in flink so I would like to have ideas on how to proceed.

--

[Fabian Hueske-2]

Hi,

that does not sound like a time window problem because there is not time-related condition to split the windows.

I think you can implement that with a GlobalWindow and a custom trigger. The documentation about global windows, triggers, and evictors [1] and this blogpost [2] might be helpful

On thing that came to my mind reading your description, are you sure that the events arrive in the correct order?

Best,

Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/windows.html
[2] http://flink.apache.org/news/2015/12/04/Introducing-windows.html

2016-11-18 1:04 GMT+01:00 Abdul Salam Shaikh <[hidden email]>:

Hi,

I have a requirement for my thesis project where I need to set the time window based on the value which I received in the event stream. 

I need to collect the values when the value starts at 0 and increments to a max value(which is unknown) till the value resets to 0 again on which a trigger is triggered which performs a calculation based on the values collected. 

DataStream<Tuple4<String,Double,String,Integer>> detectorEventStream = streamMatchedTuples

        .keyBy(Based on the first param of the tuple)

        .timeWindow(based on the second param of the tuple)

        .apply(new PhaseCalculator());

I am relatively new in flink so I would like to have ideas on how to proceed.

--

[Abdul Salam Shaikh]

Hello Mr Hueske,

Thank you for reaching out to my query.

The example stated in the documentation is the same use case for me where I am trying to build a prototype regarding a traffic metric in Germany as a part of my thesis. 

The data is received from multiple detectors and there is a field which measures the current cycle time(Umlaufsekunde) in the whole cycle.  So, as far as my analysis I haven't seen any event as of now where the previous Umlaufsekunde greater than the new one.

So if I understand you correctly, 

 .keyBy(<key selector>)
    .window(<window assigner>)  --- Global Window
    .trigger(<trigger>)         --- Indicates when the collected data is ready to be processed
    .<windowed transformation>(<window function>); --- Actual data processing

Could you point me to some repo(if you know of any) which has examples of implementation of Triggers and Windows ? 

Also, will the concept of generating timestamps/watermarks be applicable in this scenario ? 

On Fri, Nov 18, 2016 at 9:50 AM, Fabian Hueske <[hidden email]> wrote:

Hi,

that does not sound like a time window problem because there is not time-related condition to split the windows.

I think you can implement that with a GlobalWindow and a custom trigger. The documentation about global windows, triggers, and evictors [1] and this blogpost [2] might be helpful

On thing that came to my mind reading your description, are you sure that the events arrive in the correct order?

Best,

Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/windows.html
[2] http://flink.apache.org/news/2015/12/04/Introducing-windows.html

2016-11-18 1:04 GMT+01:00 Abdul Salam Shaikh <[hidden email]>:

Hi,

I have a requirement for my thesis project where I need to set the time window based on the value which I received in the event stream. 

I need to collect the values when the value starts at 0 and increments to a max value(which is unknown) till the value resets to 0 again on which a trigger is triggered which performs a calculation based on the values collected. 

DataStream<Tuple4<String,Double,String,Integer>> detectorEventStream = streamMatchedTuples

        .keyBy(Based on the first param of the tuple)

        .timeWindow(based on the second param of the tuple)

        .apply(new PhaseCalculator());

I am relatively new in flink so I would like to have ideas on how to proceed.

--

--

[Fabian Hueske-2]

Hi,

sorry for the late reply.

There is a repository [1] with an example application that uses a custom trigger [2] (though together with a TimeWIndow and not with a GlobalWindow).

I'm not aware of a repo with an example of a GlobalWIndow.

Regarding the question about timestamps and watermarks: It depends if you need them or not. If you can be sure that your data always arrives in order, you might be good without using timestamps and watermarks. Otherwise, it might make sense to set them, but then you would need to sort the data on time before you pass it into the global window.

Best,

Fabian

[1] https://github.com/dataArtisans/flink-streaming-demo
[2] https://github.com/dataArtisans/flink-streaming-demo/blob/master/src/main/scala/com/dataartisans/flink_demo/examples/EarlyArrivalCount.scala

2016-11-18 15:59 GMT+01:00 Abdul Salam Shaikh <[hidden email]>:

Hello Mr Hueske,

Thank you for reaching out to my query.

The example stated in the documentation is the same use case for me where I am trying to build a prototype regarding a traffic metric in Germany as a part of my thesis. 

The data is received from multiple detectors and there is a field which measures the current cycle time(Umlaufsekunde) in the whole cycle.  So, as far as my analysis I haven't seen any event as of now where the previous Umlaufsekunde greater than the new one.

So if I understand you correctly, 

 .keyBy(<key selector>)
    .window(<window assigner>)  --- Global Window
    .trigger(<trigger>)         --- Indicates when the collected data is ready to be processed
    .<windowed transformation>(<window function>); --- Actual data processing

Could you point me to some repo(if you know of any) which has examples of implementation of Triggers and Windows ? 

Also, will the concept of generating timestamps/watermarks be applicable in this scenario ? 

On Fri, Nov 18, 2016 at 9:50 AM, Fabian Hueske <[hidden email]> wrote:

Hi,

that does not sound like a time window problem because there is not time-related condition to split the windows.

I think you can implement that with a GlobalWindow and a custom trigger. The documentation about global windows, triggers, and evictors [1] and this blogpost [2] might be helpful

On thing that came to my mind reading your description, are you sure that the events arrive in the correct order?

Best,

Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/windows.html
[2] http://flink.apache.org/news/2015/12/04/Introducing-windows.html

2016-11-18 1:04 GMT+01:00 Abdul Salam Shaikh <[hidden email]>:

Hi,

I have a requirement for my thesis project where I need to set the time window based on the value which I received in the event stream. 

I need to collect the values when the value starts at 0 and increments to a max value(which is unknown) till the value resets to 0 again on which a trigger is triggered which performs a calculation based on the values collected. 

DataStream<Tuple4<String,Double,String,Integer>> detectorEventStream = streamMatchedTuples

        .keyBy(Based on the first param of the tuple)

        .timeWindow(based on the second param of the tuple)

        .apply(new PhaseCalculator());

I am relatively new in flink so I would like to have ideas on how to proceed.

--

--