processWindowFunction

[Antonio Saldivar Lezama]

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me 

[vino yang]

Hi Antonio,

What results do not you want to get when creating each window? 

Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala.

For more information on ProcessWindowFunction, you can refer to the official website.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 上午6:24写道：

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me 

[Antonio Saldivar Lezama]

Hi Vino

thank you for the information, actually I am using a trigger alert and processWindowFunction to send my results, but when my window slides or ends it sends again the objects and I an getting duplicated data

El jue., 16 ago. 2018 a las 22:05, vino yang (<[hidden email]>) escribió:

Hi Antonio,

What results do not you want to get when creating each window? 

Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala.

For more information on ProcessWindowFunction, you can refer to the official website.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 上午6:24写道：

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me 

[vino yang]

Hi antonio,

Yes, ProcessWindowFunction is a very low level window function. 

It allows you to access the data in the window and allows you to customize the output of the window. 

So if you use it, while giving you flexibility, you need to think about other things, which may require you to write more processing logic.

Generally speaking, sliding windows usually have some data that is repeated, but a common mode is to apply a reduce function on it to get your calculation results. 

If you only send data, there will definitely be some duplication.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 下午12:01写道：

Hi Vino

thank you for the information, actually I am using a trigger alert and processWindowFunction to send my results, but when my window slides or ends it sends again the objects and I an getting duplicated data

El jue., 16 ago. 2018 a las 22:05, vino yang (<[hidden email]>) escribió:

Hi Antonio,

What results do not you want to get when creating each window? 

Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala.

For more information on ProcessWindowFunction, you can refer to the official website.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 上午6:24写道：

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me 

[Antonio Saldivar Lezama]

hi Vino

it is possible to use global window, then set the trigger onElement comparing the element that has arrived with for example 10 mins, 20 mins and 60 mins of data?

I have rules evaluating sum of amount for 10,20 or 60 mins for the same keyed element if the same id sum like $200 total within those thresholds and count more or equals to 3 I need to be able to set some values to the object if the object does not reach those thresholds i do not set the values and keep sending the output with or without those value.

just processing the object on the fly and send output

El vie., 17 ago. 2018 a las 22:14, vino yang (<[hidden email]>) escribió:

Hi antonio,

Yes, ProcessWindowFunction is a very low level window function. 

It allows you to access the data in the window and allows you to customize the output of the window. 

So if you use it, while giving you flexibility, you need to think about other things, which may require you to write more processing logic.

Generally speaking, sliding windows usually have some data that is repeated, but a common mode is to apply a reduce function on it to get your calculation results. 

If you only send data, there will definitely be some duplication.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 下午12:01写道：

Hi Vino

thank you for the information, actually I am using a trigger alert and processWindowFunction to send my results, but when my window slides or ends it sends again the objects and I an getting duplicated data

El jue., 16 ago. 2018 a las 22:05, vino yang (<[hidden email]>) escribió:

Hi Antonio,

What results do not you want to get when creating each window? 

Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala.

For more information on ProcessWindowFunction, you can refer to the official website.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 上午6:24写道：

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me 

[vino yang]

Hi antonio,

Regarding your scenario, I think maybe you can consider using the ProcessFunction (or keyed ProcessFunction) function directly on the Stream. [1]

It can handle each of your elements with a Timer, and you can combine Flink's state API[2] to store your data.

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations

[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月19日周日 上午10:18写道：

hi Vino

it is possible to use global window, then set the trigger onElement comparing the element that has arrived with for example 10 mins, 20 mins and 60 mins of data?

I have rules evaluating sum of amount for 10,20 or 60 mins for the same keyed element if the same id sum like $200 total within those thresholds and count more or equals to 3 I need to be able to set some values to the object if the object does not reach those thresholds i do not set the values and keep sending the output with or without those value.

just processing the object on the fly and send output

El vie., 17 ago. 2018 a las 22:14, vino yang (<[hidden email]>) escribió:

Hi antonio,

Yes, ProcessWindowFunction is a very low level window function. 

It allows you to access the data in the window and allows you to customize the output of the window. 

So if you use it, while giving you flexibility, you need to think about other things, which may require you to write more processing logic.

Generally speaking, sliding windows usually have some data that is repeated, but a common mode is to apply a reduce function on it to get your calculation results. 

If you only send data, there will definitely be some duplication.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 下午12:01写道：

Hi Vino

thank you for the information, actually I am using a trigger alert and processWindowFunction to send my results, but when my window slides or ends it sends again the objects and I an getting duplicated data

El jue., 16 ago. 2018 a las 22:05, vino yang (<[hidden email]>) escribió:

Hi Antonio,

What results do not you want to get when creating each window? 

Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala.

For more information on ProcessWindowFunction, you can refer to the official website.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 上午6:24写道：

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me 

[Antonio Saldivar Lezama]

Thank you fro the references

I have now my processFunction and getting the state but now how can i do for the threshold times to group the elements and also as this is a global window, how to purge because if going to keep increasing

El dom., 19 ago. 2018 a las 8:57, vino yang (<[hidden email]>) escribió:

Hi antonio,

Regarding your scenario, I think maybe you can consider using the ProcessFunction (or keyed ProcessFunction) function directly on the Stream. [1]

It can handle each of your elements with a Timer, and you can combine Flink's state API[2] to store your data.

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations

[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月19日周日 上午10:18写道：

hi Vino

it is possible to use global window, then set the trigger onElement comparing the element that has arrived with for example 10 mins, 20 mins and 60 mins of data?

I have rules evaluating sum of amount for 10,20 or 60 mins for the same keyed element if the same id sum like $200 total within those thresholds and count more or equals to 3 I need to be able to set some values to the object if the object does not reach those thresholds i do not set the values and keep sending the output with or without those value.

just processing the object on the fly and send output

El vie., 17 ago. 2018 a las 22:14, vino yang (<[hidden email]>) escribió:

Hi antonio,

Yes, ProcessWindowFunction is a very low level window function. 

It allows you to access the data in the window and allows you to customize the output of the window. 

So if you use it, while giving you flexibility, you need to think about other things, which may require you to write more processing logic.

Generally speaking, sliding windows usually have some data that is repeated, but a common mode is to apply a reduce function on it to get your calculation results. 

If you only send data, there will definitely be some duplication.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 下午12:01写道：

Hi Vino

thank you for the information, actually I am using a trigger alert and processWindowFunction to send my results, but when my window slides or ends it sends again the objects and I an getting duplicated data

El jue., 16 ago. 2018 a las 22:05, vino yang (<[hidden email]>) escribió:

Hi Antonio,

What results do not you want to get when creating each window? 

Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala.

For more information on ProcessWindowFunction, you can refer to the official website.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 上午6:24写道：

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me 

[vino yang]

Hi antonio,

First, I suggest you use KeyedProcessFunction if you have an operation similar to keyBy. 

The implementation is similar to the Fixed window. 

You can create three state collections to determine whether the time of each element belongs to a state collection. 

At the time of the trigger, the elements in the collection are evaluated.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月20日周一 上午11:54写道：

Thank you fro the references

I have now my processFunction and getting the state but now how can i do for the threshold times to group the elements and also as this is a global window, how to purge because if going to keep increasing

El dom., 19 ago. 2018 a las 8:57, vino yang (<[hidden email]>) escribió:

Hi antonio,

Regarding your scenario, I think maybe you can consider using the ProcessFunction (or keyed ProcessFunction) function directly on the Stream. [1]

It can handle each of your elements with a Timer, and you can combine Flink's state API[2] to store your data.

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations

[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月19日周日 上午10:18写道：

hi Vino

it is possible to use global window, then set the trigger onElement comparing the element that has arrived with for example 10 mins, 20 mins and 60 mins of data?

I have rules evaluating sum of amount for 10,20 or 60 mins for the same keyed element if the same id sum like $200 total within those thresholds and count more or equals to 3 I need to be able to set some values to the object if the object does not reach those thresholds i do not set the values and keep sending the output with or without those value.

just processing the object on the fly and send output

El vie., 17 ago. 2018 a las 22:14, vino yang (<[hidden email]>) escribió:

Hi antonio,

Yes, ProcessWindowFunction is a very low level window function. 

It allows you to access the data in the window and allows you to customize the output of the window. 

So if you use it, while giving you flexibility, you need to think about other things, which may require you to write more processing logic.

Generally speaking, sliding windows usually have some data that is repeated, but a common mode is to apply a reduce function on it to get your calculation results. 

If you only send data, there will definitely be some duplication.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 下午12:01写道：

Hi Vino

thank you for the information, actually I am using a trigger alert and processWindowFunction to send my results, but when my window slides or ends it sends again the objects and I an getting duplicated data

El jue., 16 ago. 2018 a las 22:05, vino yang (<[hidden email]>) escribió:

Hi Antonio,

What results do not you want to get when creating each window? 

Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala.

For more information on ProcessWindowFunction, you can refer to the official website.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 上午6:24写道：

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me 

[Antonio Saldivar Lezama]

Hello

Thank you for the information, for some reason this KeyedProcessFunction is not found in my Flink version 1.4.2 I can only find ProcessFunction and work like this

public class TxnProcessFn extends ProcessFunction<Object,Object> {

 public void open(Configuration parameters) throws Exception {

state1 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState1", Object.class));

state2 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState2", Object.class));

state3 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState3", Object.class));

}

@Override

public void processElement(

Object obj, 

Context ctx,

Collector<Transaction> out) throws Exception {

// TODO Auto-generated method stub

Object current = state.value();

if (current == null) {

            current = new Object();

            current.id=obj.id();

   

        }

}

El lun., 20 ago. 2018 a las 2:24, vino yang (<[hidden email]>) escribió:

Hi antonio,

First, I suggest you use KeyedProcessFunction if you have an operation similar to keyBy. 

The implementation is similar to the Fixed window. 

You can create three state collections to determine whether the time of each element belongs to a state collection. 

At the time of the trigger, the elements in the collection are evaluated.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月20日周一 上午11:54写道：

Thank you fro the references

I have now my processFunction and getting the state but now how can i do for the threshold times to group the elements and also as this is a global window, how to purge because if going to keep increasing

El dom., 19 ago. 2018 a las 8:57, vino yang (<[hidden email]>) escribió:

Hi antonio,

Regarding your scenario, I think maybe you can consider using the ProcessFunction (or keyed ProcessFunction) function directly on the Stream. [1]

It can handle each of your elements with a Timer, and you can combine Flink's state API[2] to store your data.

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations

[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月19日周日 上午10:18写道：

hi Vino

it is possible to use global window, then set the trigger onElement comparing the element that has arrived with for example 10 mins, 20 mins and 60 mins of data?

I have rules evaluating sum of amount for 10,20 or 60 mins for the same keyed element if the same id sum like $200 total within those thresholds and count more or equals to 3 I need to be able to set some values to the object if the object does not reach those thresholds i do not set the values and keep sending the output with or without those value.

just processing the object on the fly and send output

El vie., 17 ago. 2018 a las 22:14, vino yang (<[hidden email]>) escribió:

Hi antonio,

Yes, ProcessWindowFunction is a very low level window function. 

It allows you to access the data in the window and allows you to customize the output of the window. 

So if you use it, while giving you flexibility, you need to think about other things, which may require you to write more processing logic.

Generally speaking, sliding windows usually have some data that is repeated, but a common mode is to apply a reduce function on it to get your calculation results. 

If you only send data, there will definitely be some duplication.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 下午12:01写道：

Hi Vino

thank you for the information, actually I am using a trigger alert and processWindowFunction to send my results, but when my window slides or ends it sends again the objects and I an getting duplicated data

El jue., 16 ago. 2018 a las 22:05, vino yang (<[hidden email]>) escribió:

Hi Antonio,

What results do not you want to get when creating each window? 

Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala.

For more information on ProcessWindowFunction, you can refer to the official website.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 上午6:24写道：

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me 

[vino yang]

Hi antonio,

Oh, if you can't use KeyedProcessFunction, then this would be a pity. 

Then you can use MapState, where Key is used to store the key of your partition. 

But I am not sure if this will achieve the effect you want.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月20日周一 下午4:32写道：

Hello

Thank you for the information, for some reason this KeyedProcessFunction is not found in my Flink version 1.4.2 I can only find ProcessFunction and work like this

public class TxnProcessFn extends ProcessFunction<Object,Object> {

 public void open(Configuration parameters) throws Exception {

state1 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState1", Object.class));

state2 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState2", Object.class));

state3 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState3", Object.class));

}

@Override

public void processElement(

Object obj, 

Context ctx,

Collector<Transaction> out) throws Exception {

// TODO Auto-generated method stub

Object current = state.value();

if (current == null) {

            current = new Object();

            current.id=obj.id();

   

        }

}

El lun., 20 ago. 2018 a las 2:24, vino yang (<[hidden email]>) escribió:

Hi antonio,

First, I suggest you use KeyedProcessFunction if you have an operation similar to keyBy. 

The implementation is similar to the Fixed window. 

You can create three state collections to determine whether the time of each element belongs to a state collection. 

At the time of the trigger, the elements in the collection are evaluated.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月20日周一 上午11:54写道：

Thank you fro the references

I have now my processFunction and getting the state but now how can i do for the threshold times to group the elements and also as this is a global window, how to purge because if going to keep increasing

El dom., 19 ago. 2018 a las 8:57, vino yang (<[hidden email]>) escribió:

Hi antonio,

Regarding your scenario, I think maybe you can consider using the ProcessFunction (or keyed ProcessFunction) function directly on the Stream. [1]

It can handle each of your elements with a Timer, and you can combine Flink's state API[2] to store your data.

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations

[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月19日周日 上午10:18写道：

hi Vino

it is possible to use global window, then set the trigger onElement comparing the element that has arrived with for example 10 mins, 20 mins and 60 mins of data?

I have rules evaluating sum of amount for 10,20 or 60 mins for the same keyed element if the same id sum like $200 total within those thresholds and count more or equals to 3 I need to be able to set some values to the object if the object does not reach those thresholds i do not set the values and keep sending the output with or without those value.

just processing the object on the fly and send output

El vie., 17 ago. 2018 a las 22:14, vino yang (<[hidden email]>) escribió:

Hi antonio,

Yes, ProcessWindowFunction is a very low level window function. 

It allows you to access the data in the window and allows you to customize the output of the window. 

So if you use it, while giving you flexibility, you need to think about other things, which may require you to write more processing logic.

Generally speaking, sliding windows usually have some data that is repeated, but a common mode is to apply a reduce function on it to get your calculation results. 

If you only send data, there will definitely be some duplication.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 下午12:01写道：

Hi Vino

thank you for the information, actually I am using a trigger alert and processWindowFunction to send my results, but when my window slides or ends it sends again the objects and I an getting duplicated data

El jue., 16 ago. 2018 a las 22:05, vino yang (<[hidden email]>) escribió:

Hi Antonio,

What results do not you want to get when creating each window? 

Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala.

For more information on ProcessWindowFunction, you can refer to the official website.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 上午6:24写道：

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me 

[Antonio Saldivar Lezama]

Maybe the usage of that function change, now I have to use it as this [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#the-keyedprocessfunction

El lun., 20 ago. 2018 a las 5:56, vino yang (<[hidden email]>) escribió:

Hi antonio,

Oh, if you can't use KeyedProcessFunction, then this would be a pity. 

Then you can use MapState, where Key is used to store the key of your partition. 

But I am not sure if this will achieve the effect you want.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月20日周一 下午4:32写道：

Hello

Thank you for the information, for some reason this KeyedProcessFunction is not found in my Flink version 1.4.2 I can only find ProcessFunction and work like this

public class TxnProcessFn extends ProcessFunction<Object,Object> {

 public void open(Configuration parameters) throws Exception {

state1 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState1", Object.class));

state2 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState2", Object.class));

state3 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState3", Object.class));

}

@Override

public void processElement(

Object obj, 

Context ctx,

Collector<Transaction> out) throws Exception {

// TODO Auto-generated method stub

Object current = state.value();

if (current == null) {

            current = new Object();

            current.id=obj.id();

   

        }

}

El lun., 20 ago. 2018 a las 2:24, vino yang (<[hidden email]>) escribió:

Hi antonio,

First, I suggest you use KeyedProcessFunction if you have an operation similar to keyBy. 

The implementation is similar to the Fixed window. 

You can create three state collections to determine whether the time of each element belongs to a state collection. 

At the time of the trigger, the elements in the collection are evaluated.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月20日周一 上午11:54写道：

Thank you fro the references

I have now my processFunction and getting the state but now how can i do for the threshold times to group the elements and also as this is a global window, how to purge because if going to keep increasing

El dom., 19 ago. 2018 a las 8:57, vino yang (<[hidden email]>) escribió:

Hi antonio,

Regarding your scenario, I think maybe you can consider using the ProcessFunction (or keyed ProcessFunction) function directly on the Stream. [1]

It can handle each of your elements with a Timer, and you can combine Flink's state API[2] to store your data.

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations

[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月19日周日 上午10:18写道：

hi Vino

it is possible to use global window, then set the trigger onElement comparing the element that has arrived with for example 10 mins, 20 mins and 60 mins of data?

I have rules evaluating sum of amount for 10,20 or 60 mins for the same keyed element if the same id sum like $200 total within those thresholds and count more or equals to 3 I need to be able to set some values to the object if the object does not reach those thresholds i do not set the values and keep sending the output with or without those value.

just processing the object on the fly and send output

El vie., 17 ago. 2018 a las 22:14, vino yang (<[hidden email]>) escribió:

Hi antonio,

Yes, ProcessWindowFunction is a very low level window function. 

It allows you to access the data in the window and allows you to customize the output of the window. 

So if you use it, while giving you flexibility, you need to think about other things, which may require you to write more processing logic.

Generally speaking, sliding windows usually have some data that is repeated, but a common mode is to apply a reduce function on it to get your calculation results. 

If you only send data, there will definitely be some duplication.

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 下午12:01写道：

Hi Vino

thank you for the information, actually I am using a trigger alert and processWindowFunction to send my results, but when my window slides or ends it sends again the objects and I an getting duplicated data

El jue., 16 ago. 2018 a las 22:05, vino yang (<[hidden email]>) escribió:

Hi Antonio,

What results do not you want to get when creating each window? 

Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala.

For more information on ProcessWindowFunction, you can refer to the official website.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction

Thanks, vino.

antonio saldivar <[hidden email]> 于2018年8月17日周五 上午6:24写道：

Hello

I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function  and send the object to the next step but I don't want results every time a window is creating 

if anyone has a good example on this that can help me