How does Flink handle shorted lived keyed streams

[narasimha]

Hi,

Belos is the use case.

Have a stream of transaction events, success/failure of a transaction can be determined by those events. 

Partitioning stream by transaction id and applying CEP to determine the success/failure of a transaction.

Each transaction keyed stream is valid only until the final status is found. Which can end up having large inactive keyed streams in the system. 

Know that using keygroup flink distributes the keyedstream to tasks based on it, but still there will be a large set of inactive keys.

Does this have any side effects? If so what has to be done to overcome humongous keyed streams?

--

[Xintong Song]

I believe what you are looking for is the State TTL [1][2].

On Wed, Dec 23, 2020 at 11:57 PM narasimha <[hidden email]> wrote:

Hi,

Belos is the use case.

Have a stream of transaction events, success/failure of a transaction can be determined by those events. 

Partitioning stream by transaction id and applying CEP to determine the success/failure of a transaction.

Each transaction keyed stream is valid only until the final status is found. Which can end up having large inactive keyed streams in the system. 

Know that using keygroup flink distributes the keyedstream to tasks based on it, but still there will be a large set of inactive keys.

Does this have any side effects? If so what has to be done to overcome humongous keyed streams?

--

A.Narasimha Swamy

[narasimha]

Thanks Xintong.

I'll check it out and get back to you.

On Thu, Dec 24, 2020 at 1:30 PM Xintong Song <[hidden email]> wrote:

I believe what you are looking for is the State TTL [1][2].

Thank you~

Xintong Song

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl

[2] https://ci.apache.org/projects/flink/flink-docs-stabledev/table/config.html#table-exec-state-ttl

On Wed, Dec 23, 2020 at 11:57 PM narasimha <[hidden email]> wrote:

Hi,

Belos is the use case.

Have a stream of transaction events, success/failure of a transaction can be determined by those events. 

Partitioning stream by transaction id and applying CEP to determine the success/failure of a transaction.

Each transaction keyed stream is valid only until the final status is found. Which can end up having large inactive keyed streams in the system. 

Know that using keygroup flink distributes the keyedstream to tasks based on it, but still there will be a large set of inactive keys.

Does this have any side effects? If so what has to be done to overcome humongous keyed streams?

--

A.Narasimha Swamy

--

[narasimha]

It is not solving the problem.

I could see the memory keep increasing, resulting in a lot of high GCs.

There could be a memory leak, just want to know how to know if older keps are skill alive, even after the pattern has been satisfied or within range of the pattern has expired. 

Can someone suggest how to proceed further. 

[Matthias]

Hi narashima,

not sure whether this fits your use case, but have you considered creating a savepoint and analyzing it using the State Processor API [1]?

Best,
Matthias

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html#state-processor-api

On Wed, Feb 10, 2021 at 6:08 PM narasimha <[hidden email]> wrote:

It is not solving the problem.

I could see the memory keep increasing, resulting in a lot of high GCs.

There could be a memory leak, just want to know how to know if older keps are skill alive, even after the pattern has been satisfied or within range of the pattern has expired. 

Can someone suggest how to proceed further. 

[narasimha]

Thanks Matthias, I think it will help to find out what all live keys are present. 

Let me check and revert back on the thread. 

On Wed, Feb 10, 2021 at 10:46 PM Matthias Pohl <[hidden email]> wrote:

Hi narashima,

not sure whether this fits your use case, but have you considered creating a savepoint and analyzing it using the State Processor API [1]?

Best,
Matthias

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html#state-processor-api

On Wed, Feb 10, 2021 at 6:08 PM narasimha <[hidden email]> wrote:

It is not solving the problem.

I could see the memory keep increasing, resulting in a lot of high GCs.

There could be a memory leak, just want to know how to know if older keps are skill alive, even after the pattern has been satisfied or within range of the pattern has expired. 

Can someone suggest how to proceed further. 

--