Re: state access causing segmentation fault

Posted by Dawid Wysakowicz-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/state-access-causing-segmentation-fault-tp38562p38564.html

Hi,

It should be absolutely fine to use multiple state objects. I am not aware of any limits to that. A minimal, reproducible example would definitely be helpful. For those kind of exceptions, I'd look into the serializers you use. Other than that I cannot think of an obvious reason for that kind of exceptions.

Best,

Dawid

On 08/10/2020 12:12, Colletta, Edward wrote:

Using Flink 1.9.2, Java, FsStateBackend.  Running Session cluster on EC2 instances.

 

I have a KeyedProcessFunction that is causing a segmentation fault, crashing the flink task manager.  The seems to be caused by using 3 State variables in the operator.  The crash happens consistently after some load is processed.

This is the second time I have encountered this.   The first time I had 3 ValueState variables, this time I had 2 ValueState variables and a MapState variable.  Both times the error was alleviated by removing one of the state variables.

This time I replaced the 2 valueState variables with a Tuple2 of the types of the individual variables.   I can try to put together a minimal example, but I was wondering if anyone has encountered this problem.

 

Are there any documented limits of the number of state variables 1 operator can use?

 

For background the reason I use multiple state variables is the operator is processing 2 types of inputs, Left and Right.  When Left is received it is put it into a PriorityQueue. When the Right type is received I put that into a ring buffer.

I replaced the PriorityQueue with a queue of Ids and MapState to hold the elements.  So I have Left stored in a queue ValueState variable and MapState variable, and Right is stored in the ring buffer ValueState variable.