Re: Slow flink checkpoint

Posted by Fabian Hueske-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Slow-flink-checkpoint-tp18946p18991.html

Hi,

Yes, you cannot start a separate thread to cleanup the state.
State is managed by Flink and can only be accessed at certain points in time when the user code is called.

If you are using event time, another trick you could play is to only register all timers on (currentWatermark + 1).

That will cause the trigger to fire whenever the watermark advances. You could store all relevant timestamps in a ListState and act on all timers that are less than the currentWatermark.

Also, since there is only a single timer per timestamp (currentWM + 1) there will be only one watermark per key.

Best, Fabian

2018-03-16 13:56 GMT+01:00 林德强 <[hidden email]>:

Hi Fabian ,

   Reduce the number of timers is a good idea. 

   But in my application the timer is different from the key  registered follow the keyBy . May be it can't work with an upper and lower bound.

   I try modify the flink resource and start a thread to clean the expired keyed sate, but it doesn't work well because of concurrency issues.

Best,

Deqiang

2018-03-16 16:03 GMT+08:00 Fabian Hueske <[hidden email]>:

Hi,

AFAIK, that's not possible.
The only "solution" is to reduce the number of timers. Whether that's possible or not, depends on the application.

For example, if you use timers to clean up state, you can work with an upper and lower bound and only register one timer for each (upper - lower) interval.

Best, Fabian

2018-03-16 3:11 GMT+01:00 林德强 <[hidden email]>:

Hi,

I'm run a job on Flink streaming. I found with the increase in the number of  'InternalTimer' object the checkpoint more and more slowly. Is there any way to solve this problem ? such as make the "timeServiceManager" snapshot async.

Thanks