General guidance

[Almeida, Julius]

Hi Team,

 

My streaming pipeline is based on beam & running using flink runner with rocksdb as state backend.

 

Over time I am  seeing memory spike & after giving a look at heap dump, I am seeing records in  ‘__StatefulParDoGcTimerId’ which seems to be never cleaned.

 

Found this jira https://issues.apache.org/jira/browse/BEAM-8212 describing the issue I believe I am facing.

 

Any pointers would be helpful in identifying possible solution.

 

Thanks,

Julius

[Kenneth Knowles]

This is a Beam issue indeed, though it is an issue with the FlinkRunner. So I think I will BCC the Flink list.

You may be in one of the following situations:

 - These timers should not be viewed as distinct by the runner, but deduped, per https://issues.apache.org/jira/browse/BEAM-8212#comment-16946013

 - There is a different problem if you have an unbounded key space with windows that never expire, since then there are unbounded numbers of truly distinct (but irrelevant) timers. That is also the responsibility of the runner to simply not set timers that can never fire.

Kenn

On Thu, Mar 25, 2021 at 11:57 AM Almeida, Julius <[hidden email]> wrote:

Hi Team,

 

My streaming pipeline is based on beam & running using flink runner with rocksdb as state backend.

 

Over time I am  seeing memory spike & after giving a look at heap dump, I am seeing records in  ‘__StatefulParDoGcTimerId’ which seems to be never cleaned.

 

Found this jira https://issues.apache.org/jira/browse/BEAM-8212 describing the issue I believe I am facing.

 

Any pointers would be helpful in identifying possible solution.

 

Thanks,

Julius