Statefun: cancel "sendAfter"

[Stephan Pelikan]

Hi,

 

I think about using „sendAfter“ to implement some kind of timer functionality. I’m wondering if there is no possibility to cancel delayed sent message!

 

In my use case it is possible that intermediate events make the delayed message obsolete. In some cases the statefun of that certain ID is cleared (clear all state variables) and does not exist anymore. In other cases the statefun of that ID still exists (and its state). In the latter case I could ignore the delayed message, but what about those statefun which do not exist anymore?

 

Additionally there can be millions of delayed messages which I do not need any more and some delays are also hours, days or even months. I don’t want to pollute my state with this because it will inflate the size of my checkpoints.

 

There are no hints in the docs (https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/java.html#sending-delayed-messages) how those situations are treated. I found in the Flink’s docs that timers of keyed processors can be deleted. As far as I know statefuns are based on those processors, so I hope that there is something about it. I hope someone can clarify what I can expect and how those situations are handled internally.

 

Thanks,

Stephan

[Tzu-Li (Gordon) Tai]

Hi,

You are right, currently StateFun does not support deleting a scheduled delayed message.

StateFun supports delayed messages by building on top of two Flink constructs: 1) registering processing time timers, and 2) buffering the message payload to be sent in state.

The delayed messages are kept in the Flink state of the sending operator, and timers are registered on the sending operator as well. So technically, there doesn't seem to be a blocker for deleting a delayed message and its associated timer, if it hasn't been sent yet.

Can you maybe open a JIRA ticket for this, so we have something that tracks it?
Also cc'ing Igal, who might have more comments on whether supporting this makes sense.

Cheers,
Gordon

On Wed, Feb 3, 2021 at 3:51 AM Stephan Pelikan <[hidden email]> wrote:

Hi,

 

I think about using „sendAfter“ to implement some kind of timer functionality. I’m wondering if there is no possibility to cancel delayed sent message!

 

In my use case it is possible that intermediate events make the delayed message obsolete. In some cases the statefun of that certain ID is cleared (clear all state variables) and does not exist anymore. In other cases the statefun of that ID still exists (and its state). In the latter case I could ignore the delayed message, but what about those statefun which do not exist anymore?

 

Additionally there can be millions of delayed messages which I do not need any more and some delays are also hours, days or even months. I don’t want to pollute my state with this because it will inflate the size of my checkpoints.

 

There are no hints in the docs (https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/java.html#sending-delayed-messages) how those situations are treated. I found in the Flink’s docs that timers of keyed processors can be deleted. As far as I know statefuns are based on those processors, so I hope that there is something about it. I hope someone can clarify what I can expect and how those situations are handled internally.

 

Thanks,

Stephan

[Stephan Pelikan]

Hi,

 

thank you Gordon for clarification. My use-case is processing business events of customers. Those events are triggered by ourself or by the customer depending of what’s the current state of the ongoing customer’s business use-case. We need to monitor delayed/missing business events which belong to previous events. For example: the customer has to confirm something we did. Depending on what it is the confirmation has to be within hours, days or even months. If there is a delay we need to know. But if the customer confirms in time we want to cleanup to keep the state small.

 

I dug a little bit into the code. May I create an issue to discuss my ideas?

 

Cheers,

Stephan

 

 

Von: Tzu-Li (Gordon) Tai <[hidden email]>
Gesendet: Mittwoch, 3. Februar 2021 07:58
An: Stephan Pelikan <[hidden email]>
Cc: [hidden email]; Igal Shilman <[hidden email]>
Betreff: Re: Statefun: cancel "sendAfter"

 

Hi,

You are right, currently StateFun does not support deleting a scheduled delayed message.

StateFun supports delayed messages by building on top of two Flink constructs: 1) registering processing time timers, and 2) buffering the message payload to be sent in state.

The delayed messages are kept in the Flink state of the sending operator, and timers are registered on the sending operator as well. So technically, there doesn't seem to be a blocker for deleting a delayed message and its associated timer, if it hasn't been sent yet.

Can you maybe open a JIRA ticket for this, so we have something that tracks it?
Also cc'ing Igal, who might have more comments on whether supporting this makes sense.

Cheers,
Gordon

 

On Wed, Feb 3, 2021 at 3:51 AM Stephan Pelikan <[hidden email]> wrote:

Hi,

 

I think about using „sendAfter“ to implement some kind of timer functionality. I’m wondering if there is no possibility to cancel delayed sent message!

 

In my use case it is possible that intermediate events make the delayed message obsolete. In some cases the statefun of that certain ID is cleared (clear all state variables) and does not exist anymore. In other cases the statefun of that ID still exists (and its state). In the latter case I could ignore the delayed message, but what about those statefun which do not exist anymore?

 

Additionally there can be millions of delayed messages which I do not need any more and some delays are also hours, days or even months. I don’t want to pollute my state with this because it will inflate the size of my checkpoints.

 

There are no hints in the docs (https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/java.html#sending-delayed-messages) how those situations are treated. I found in the Flink’s docs that timers of keyed processors can be deleted. As far as I know statefuns are based on those processors, so I hope that there is something about it. I hope someone can clarify what I can expect and how those situations are handled internally.

 

Thanks,

Stephan

[Tzu-Li (Gordon) Tai]

Hi Stephan,

Thanks for providing the details of the use case! It does indeed sound like being able to delete scheduled delayed messages would help here.

And yes, please do proceed with creating an issue. As for details on the implementation, we can continue to discuss that on the JIRA.

Cheers,
Gordon

On Wed, Feb 3, 2021 at 3:43 PM Stephan Pelikan <[hidden email]> wrote:

Hi,

 

thank you Gordon for clarification. My use-case is processing business events of customers. Those events are triggered by ourself or by the customer depending of what’s the current state of the ongoing customer’s business use-case. We need to monitor delayed/missing business events which belong to previous events. For example: the customer has to confirm something we did. Depending on what it is the confirmation has to be within hours, days or even months. If there is a delay we need to know. But if the customer confirms in time we want to cleanup to keep the state small.

 

I dug a little bit into the code. May I create an issue to discuss my ideas?

 

Cheers,

Stephan

 

 

Von: Tzu-Li (Gordon) Tai <[hidden email]>
Gesendet: Mittwoch, 3. Februar 2021 07:58
An: Stephan Pelikan <[hidden email]>
Cc: [hidden email]; Igal Shilman <[hidden email]>
Betreff: Re: Statefun: cancel "sendAfter"

 

Hi,

You are right, currently StateFun does not support deleting a scheduled delayed message.

StateFun supports delayed messages by building on top of two Flink constructs: 1) registering processing time timers, and 2) buffering the message payload to be sent in state.

The delayed messages are kept in the Flink state of the sending operator, and timers are registered on the sending operator as well. So technically, there doesn't seem to be a blocker for deleting a delayed message and its associated timer, if it hasn't been sent yet.

Can you maybe open a JIRA ticket for this, so we have something that tracks it?
Also cc'ing Igal, who might have more comments on whether supporting this makes sense.

Cheers,
Gordon

 

On Wed, Feb 3, 2021 at 3:51 AM Stephan Pelikan <[hidden email]> wrote:

Hi,

 

I think about using „sendAfter“ to implement some kind of timer functionality. I’m wondering if there is no possibility to cancel delayed sent message!

 

In my use case it is possible that intermediate events make the delayed message obsolete. In some cases the statefun of that certain ID is cleared (clear all state variables) and does not exist anymore. In other cases the statefun of that ID still exists (and its state). In the latter case I could ignore the delayed message, but what about those statefun which do not exist anymore?

 

Additionally there can be millions of delayed messages which I do not need any more and some delays are also hours, days or even months. I don’t want to pollute my state with this because it will inflate the size of my checkpoints.

 

There are no hints in the docs (https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/java.html#sending-delayed-messages) how those situations are treated. I found in the Flink’s docs that timers of keyed processors can be deleted. As far as I know statefuns are based on those processors, so I hope that there is something about it. I hope someone can clarify what I can expect and how those situations are handled internally.

 

Thanks,

Stephan

[Stephan Pelikan]

Hi Gorden,

 

here is the link, if anyone else is also interested: https://issues.apache.org/jira/browse/FLINK-21308

 

Cheers,

Stephan

 

Von: Tzu-Li (Gordon) Tai <[hidden email]>
Gesendet: Freitag, 5. Februar 2021 12:58
An: Stephan Pelikan <[hidden email]>
Cc: [hidden email]; Igal Shilman <[hidden email]>
Betreff: Re: Statefun: cancel "sendAfter"

 

Hi Stephan,

Thanks for providing the details of the use case! It does indeed sound like being able to delete scheduled delayed messages would help here.

And yes, please do proceed with creating an issue. As for details on the implementation, we can continue to discuss that on the JIRA.

Cheers,
Gordon

 

On Wed, Feb 3, 2021 at 3:43 PM Stephan Pelikan <[hidden email]> wrote:

Hi,

 

thank you Gordon for clarification. My use-case is processing business events of customers. Those events are triggered by ourself or by the customer depending of what’s the current state of the ongoing customer’s business use-case. We need to monitor delayed/missing business events which belong to previous events. For example: the customer has to confirm something we did. Depending on what it is the confirmation has to be within hours, days or even months. If there is a delay we need to know. But if the customer confirms in time we want to cleanup to keep the state small.

 

I dug a little bit into the code. May I create an issue to discuss my ideas?

 

Cheers,

Stephan

 

 

Von: Tzu-Li (Gordon) Tai <[hidden email]>
Gesendet: Mittwoch, 3. Februar 2021 07:58
An: Stephan Pelikan <[hidden email]>
Cc: [hidden email]; Igal Shilman <[hidden email]>
Betreff: Re: Statefun: cancel "sendAfter"

 

Hi,

You are right, currently StateFun does not support deleting a scheduled delayed message.

StateFun supports delayed messages by building on top of two Flink constructs: 1) registering processing time timers, and 2) buffering the message payload to be sent in state.

The delayed messages are kept in the Flink state of the sending operator, and timers are registered on the sending operator as well. So technically, there doesn't seem to be a blocker for deleting a delayed message and its associated timer, if it hasn't been sent yet.

Can you maybe open a JIRA ticket for this, so we have something that tracks it?
Also cc'ing Igal, who might have more comments on whether supporting this makes sense.

Cheers,
Gordon

 

On Wed, Feb 3, 2021 at 3:51 AM Stephan Pelikan <[hidden email]> wrote:

Hi,

 

I think about using „sendAfter“ to implement some kind of timer functionality. I’m wondering if there is no possibility to cancel delayed sent message!

 

In my use case it is possible that intermediate events make the delayed message obsolete. In some cases the statefun of that certain ID is cleared (clear all state variables) and does not exist anymore. In other cases the statefun of that ID still exists (and its state). In the latter case I could ignore the delayed message, but what about those statefun which do not exist anymore?

 

Additionally there can be millions of delayed messages which I do not need any more and some delays are also hours, days or even months. I don’t want to pollute my state with this because it will inflate the size of my checkpoints.

 

There are no hints in the docs (https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/java.html#sending-delayed-messages) how those situations are treated. I found in the Flink’s docs that timers of keyed processors can be deleted. As far as I know statefuns are based on those processors, so I hope that there is something about it. I hope someone can clarify what I can expect and how those situations are handled internally.

 

Thanks,

Stephan