Re: State size increasing exponentially in Flink v1.9

Posted by Yun Tang on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/State-size-increasing-exponentially-in-Flink-v1-9-tp42529p42583.html

Hi,

If using RocksDB state backend, why it would occur `CopyOnWriteStateMap`?

CopyOnWriteStateMap should only exist in heap based state-backend.

Best

Yun Tang

---

From: Chesnay Schepler <[hidden email]>
Sent: Friday, March 26, 2021 18:45
To: Almeida, Julius <[hidden email]>; [hidden email] <[hidden email]>
Subject: Re: State size increasing exponentially in Flink v1.9

 

Could you show us how you interact with the map state (ideally the full code of your function that accesses the state)?

On 3/25/2021 1:13 AM, Almeida, Julius wrote:

Hey,

Hope you all are doing well!

 

I am using flink v1.9 with RocksDBStateBackend, but over time the state size is increasing exponentially.

 

I am using MapState in my project & seeing memory spike, after looking at heap dump I see duplicates in it.

 

I also have logic added to remove expired events form the MapState

Eg.: MapState.remove(key)

 

Can anyone give me pointers to find more details on it.

 

Heap Dump pointed to https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/CopyOnWriteStateMap.java#L811

 

Thanks,

Julius