Queryable State race condition or serialization errors?

[burgesschen]

Hi Guys,

I observed some strange behaviors while using Queryable state with Flink
1.6.2. Here is the story:

My state is of type MapState[String, Map[String, String]]. the inner map is
frequently updated. Upon querying, sometimes the returned inner map can miss
some fields. What's more, sometimes the returned inner map has the values
assigned to other keys!

Changing the type to MapState[String, String] seem to solve the problem.

The code is a little too deep to dig into. But my guess is that when the
state is being updated and queried at the same time, there can be a race
condition and cause data corruption. Please let me know if you have a better
idea what could be happening. Much appreciated!

Best,
Burgess Chen  



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Gary Yao-4]

Hi Burgess Chen,

If you are using MemoryStateBackend or FsStateBackend, you can observe race
conditions on the state objects. However, the RocksDBStateBackend should be
safe from these issues [1].

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/queryable_state.html

On Tue, May 21, 2019 at 5:06 AM burgesschen <[hidden email]> wrote:

Hi Guys,

I observed some strange behaviors while using Queryable state with Flink
1.6.2. Here is the story:

My state is of type MapState[String, Map[String, String]]. the inner map is
frequently updated. Upon querying, sometimes the returned inner map can miss
some fields. What's more, sometimes the returned inner map has the values
assigned to other keys!

Changing the type to MapState[String, String] seem to solve the problem.

The code is a little too deep to dig into. But my guess is that when the
state is being updated and queried at the same time, there can be a race
condition and cause data corruption. Please let me know if you have a better
idea what could be happening. Much appreciated!

Best,
Burgess Chen 



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[burgesschen]

Hi Gary.

Thanks for the reply. I am using RocksDBStateBackend though.

Best,
Chen



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Tzu-Li (Gordon) Tai]

Hi Burgess,

Would you be able to provide a minimal project that can reproduce your error?
That would help a lot with figuring out the issue.
If you prefer to share that only privately, please feel free to send me a private email with the details.
Another thing you can do is set logging level to "DEBUG". We have some checks enabled at that level to see if serializers are being concurrently used across threads.

Cheers,
Gordon

On Tue, May 21, 2019 at 9:59 PM burgesschen <[hidden email]> wrote:

Hi Gary.

Thanks for the reply. I am using RocksDBStateBackend though.

Best,
Chen



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/