How to trigger a function on the state periodically?

[anna stax]

Hi all,

I need to trigger a function once every day to read the state and create kafka events and also remove some records from state if they are too old.

Is there a way to do this? I am new to Flink, appreciate any feedback and suggestions.

Thanks

Anna

[Hequn Cheng]

Hi anna,

> I need to trigger a function once every day

If you want to trigger by the function itself, you can use the Timer[1]. Both types of timers (processing-time and event-time) are internally maintained by the TimerService, and onTimer() method will be called once a timer fires.

If you want to trigger the function of different parallelism synchronously, then the broadcast state[2] may be helpful.

Hope this helps.

Hequn

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/process_function.html#timers

[2] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/broadcast_state.html

On Tue, Jul 10, 2018 at 7:47 AM, anna stax <[hidden email]> wrote:

Hi all,

I need to trigger a function once every day to read the state and create kafka events and also remove some records from state if they are too old.

Is there a way to do this? I am new to Flink, appreciate any feedback and suggestions.

Thanks

Anna

[anna stax]

Thanks Hequn, for the links.

This is my use case..

When there is no user activity for n weeks, I need to send a Notification to user. 

The activity stream is usually very high volume for most users. 

I thought it is not a good idea to use windowing for this, because of the stream volume and window size.

I want to store in the state, for every user the last activity date and process them once daily.

I want to make sure I am heading in the right direction. Thank you for your suggestions.

-Anna

On Mon, Jul 9, 2018 at 7:16 PM, Hequn Cheng <[hidden email]> wrote:

Hi anna,

> I need to trigger a function once every day

If you want to trigger by the function itself, you can use the Timer[1]. Both types of timers (processing-time and event-time) are internally maintained by the TimerService, and onTimer() method will be called once a timer fires.

If you want to trigger the function of different parallelism synchronously, then the broadcast state[2] may be helpful.

Hope this helps.

Hequn

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/process_function.html#timers

[2] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/broadcast_state.html

On Tue, Jul 10, 2018 at 7:47 AM, anna stax <[hidden email]> wrote:

Hi all,

I need to trigger a function once every day to read the state and create kafka events and also remove some records from state if they are too old.

Is there a way to do this? I am new to Flink, appreciate any feedback and suggestions.

Thanks

Anna

[Hequn Cheng]

Hi anna,

According to your description, I think we can use the Timer to solve your problem. The TimerService deduplicates timers per key and timestamp. Also, note that a large number of timers can significantly increase checkpointing time.

On Tue, Jul 10, 2018 at 11:38 AM, anna stax <[hidden email]> wrote:

Thanks Hequn, for the links.

This is my use case..

When there is no user activity for n weeks, I need to send a Notification to user. 

The activity stream is usually very high volume for most users. 

I thought it is not a good idea to use windowing for this, because of the stream volume and window size.

I want to store in the state, for every user the last activity date and process them once daily.

I want to make sure I am heading in the right direction. Thank you for your suggestions.

-Anna

On Mon, Jul 9, 2018 at 7:16 PM, Hequn Cheng <[hidden email]> wrote:

Hi anna,

> I need to trigger a function once every day

If you want to trigger by the function itself, you can use the Timer[1]. Both types of timers (processing-time and event-time) are internally maintained by the TimerService, and onTimer() method will be called once a timer fires.

If you want to trigger the function of different parallelism synchronously, then the broadcast state[2] may be helpful.

Hope this helps.

Hequn

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/process_function.html#timers

[2] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/broadcast_state.html

On Tue, Jul 10, 2018 at 7:47 AM, anna stax <[hidden email]> wrote:

Hi all,

I need to trigger a function once every day to read the state and create kafka events and also remove some records from state if they are too old.

Is there a way to do this? I am new to Flink, appreciate any feedback and suggestions.

Thanks

Anna

[anna stax]

Thanks Hequn. I think so too, the large number of timers could be a problem. 

On Mon, Jul 9, 2018 at 10:23 PM, Hequn Cheng <[hidden email]> wrote:

Hi anna,

According to your description, I think we can use the Timer to solve your problem. The TimerService deduplicates timers per key and timestamp. Also, note that a large number of timers can significantly increase checkpointing time.

On Tue, Jul 10, 2018 at 11:38 AM, anna stax <[hidden email]> wrote:

Thanks Hequn, for the links.

This is my use case..

When there is no user activity for n weeks, I need to send a Notification to user. 

The activity stream is usually very high volume for most users. 

I thought it is not a good idea to use windowing for this, because of the stream volume and window size.

I want to store in the state, for every user the last activity date and process them once daily.

I want to make sure I am heading in the right direction. Thank you for your suggestions.

-Anna

On Mon, Jul 9, 2018 at 7:16 PM, Hequn Cheng <[hidden email]> wrote:

Hi anna,

> I need to trigger a function once every day

If you want to trigger by the function itself, you can use the Timer[1]. Both types of timers (processing-time and event-time) are internally maintained by the TimerService, and onTimer() method will be called once a timer fires.

If you want to trigger the function of different parallelism synchronously, then the broadcast state[2] may be helpful.

Hope this helps.

Hequn

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/process_function.html#timers

[2] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/broadcast_state.html

On Tue, Jul 10, 2018 at 7:47 AM, anna stax <[hidden email]> wrote:

Hi all,

I need to trigger a function once every day to read the state and create kafka events and also remove some records from state if they are too old.

Is there a way to do this? I am new to Flink, appreciate any feedback and suggestions.

Thanks

Anna

[Hequn Cheng]

Hi, 

It depends on how many different users. In most cases, the performance will be fine. I think it worth to give a try. :-)

Of course, there are ways to reduce the number of timers, for example keyBy(userId%1024), and use a MapState to store different users for the same group.

On Tue, Jul 10, 2018 at 1:54 PM, anna stax <[hidden email]> wrote:

Thanks Hequn. I think so too, the large number of timers could be a problem. 

On Mon, Jul 9, 2018 at 10:23 PM, Hequn Cheng <[hidden email]> wrote:

Hi anna,

According to your description, I think we can use the Timer to solve your problem. The TimerService deduplicates timers per key and timestamp. Also, note that a large number of timers can significantly increase checkpointing time.

On Tue, Jul 10, 2018 at 11:38 AM, anna stax <[hidden email]> wrote:

Thanks Hequn, for the links.

This is my use case..

When there is no user activity for n weeks, I need to send a Notification to user. 

The activity stream is usually very high volume for most users. 

I thought it is not a good idea to use windowing for this, because of the stream volume and window size.

I want to store in the state, for every user the last activity date and process them once daily.

I want to make sure I am heading in the right direction. Thank you for your suggestions.

-Anna

On Mon, Jul 9, 2018 at 7:16 PM, Hequn Cheng <[hidden email]> wrote:

Hi anna,

> I need to trigger a function once every day

If you want to trigger by the function itself, you can use the Timer[1]. Both types of timers (processing-time and event-time) are internally maintained by the TimerService, and onTimer() method will be called once a timer fires.

If you want to trigger the function of different parallelism synchronously, then the broadcast state[2] may be helpful.

Hope this helps.

Hequn

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/process_function.html#timers

[2] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/broadcast_state.html

On Tue, Jul 10, 2018 at 7:47 AM, anna stax <[hidden email]> wrote:

Hi all,

I need to trigger a function once every day to read the state and create kafka events and also remove some records from state if they are too old.

Is there a way to do this? I am new to Flink, appreciate any feedback and suggestions.

Thanks

Anna

[anna stax]

sure. I will go ahead with this for now. Thanks for your suggestions.

On Mon, Jul 9, 2018 at 11:10 PM, Hequn Cheng <[hidden email]> wrote:

Hi, 

It depends on how many different users. In most cases, the performance will be fine. I think it worth to give a try. :-)

Of course, there are ways to reduce the number of timers, for example keyBy(userId%1024), and use a MapState to store different users for the same group.

On Tue, Jul 10, 2018 at 1:54 PM, anna stax <[hidden email]> wrote:

Thanks Hequn. I think so too, the large number of timers could be a problem. 

On Mon, Jul 9, 2018 at 10:23 PM, Hequn Cheng <[hidden email]> wrote:

Hi anna,

According to your description, I think we can use the Timer to solve your problem. The TimerService deduplicates timers per key and timestamp. Also, note that a large number of timers can significantly increase checkpointing time.

On Tue, Jul 10, 2018 at 11:38 AM, anna stax <[hidden email]> wrote:

Thanks Hequn, for the links.

This is my use case..

When there is no user activity for n weeks, I need to send a Notification to user. 

The activity stream is usually very high volume for most users. 

I thought it is not a good idea to use windowing for this, because of the stream volume and window size.

I want to store in the state, for every user the last activity date and process them once daily.

I want to make sure I am heading in the right direction. Thank you for your suggestions.

-Anna

On Mon, Jul 9, 2018 at 7:16 PM, Hequn Cheng <[hidden email]> wrote:

Hi anna,

> I need to trigger a function once every day

If you want to trigger by the function itself, you can use the Timer[1]. Both types of timers (processing-time and event-time) are internally maintained by the TimerService, and onTimer() method will be called once a timer fires.

If you want to trigger the function of different parallelism synchronously, then the broadcast state[2] may be helpful.

Hope this helps.

Hequn

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/process_function.html#timers

[2] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/broadcast_state.html

On Tue, Jul 10, 2018 at 7:47 AM, anna stax <[hidden email]> wrote:

Hi all,

I need to trigger a function once every day to read the state and create kafka events and also remove some records from state if they are too old.

Is there a way to do this? I am new to Flink, appreciate any feedback and suggestions.

Thanks

Anna