Is it possible to specify max process memory in flink 1.8.2, similar to what is possible in flink 1.11

[Bariša]

I'm running flink 1.8.2 in a container, and under heavy load, container gets OOM from the kernel.

I'm guessing that that reason for the kernel OOM is large size of the off-heap memory. Is there a way I can limit it in flink 1.8.2?

I can see that newer version of flink has a config param, checking here is it possible to do something similar in flink 1.8.2, without a flink upgrade?

Cheers,

Barisa

[Bariša]

Small update:

 we believe that the off heap memory is used by the parquet writer ( used in sink to write to S3 )

On Wed, 24 Feb 2021 at 23:25, Bariša <[hidden email]> wrote:

I'm running flink 1.8.2 in a container, and under heavy load, container gets OOM from the kernel.

I'm guessing that that reason for the kernel OOM is large size of the off-heap memory. Is there a way I can limit it in flink 1.8.2?

I can see that newer version of flink has a config param, checking here is it possible to do something similar in flink 1.8.2, without a flink upgrade?

Cheers,

Barisa

[Timo Walther]

Hi Barisa,

by looking at the 1.8 documentation [1] it was possible to configure the
off heap memory as well. Also other memory options were already present.
So I don't think that you need an upgrade to 1.11 immediately. Please
let us know if you could fix your problem, otherwise we can try to loop
in other people that should know better.

Regards,
Timo


[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-memory-off-heap



On 25.02.21 15:50, Bariša wrote:

> Small update:
>   we believe that the off heap memory is used by the parquet writer (
> used in sink to write to S3 )
>
> On Wed, 24 Feb 2021 at 23:25, Bariša <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I'm running flink 1.8.2 in a container, and under heavy load,
>     container gets OOM from the kernel.
>     I'm guessing that that reason for the kernel OOM is large size of
>     the off-heap memory. Is there a way I can limit it in flink 1.8.2?
>
>     I can see that newer version of flink has a config param, checking
>     here is it possible to do something similar in flink 1.8.2, without
>     a flink upgrade?
>
>     Cheers,
>     Barisa
>

[Matthias]

Hi Bariša,

have you had the chance to analyze the memory usage in more detail? An OutOfMemoryError might be an indication for some memory leak which should be solved instead of lowering some memory configuration parameters. Or is it that the off-heap memory is not actually used but blocks the JVM from using the allocated memory for other things?

Best,
Matthias

On Fri, Feb 26, 2021 at 10:05 AM Timo Walther <[hidden email]> wrote:

Hi Barisa,

by looking at the 1.8 documentation [1] it was possible to configure the
off heap memory as well. Also other memory options were already present.
So I don't think that you need an upgrade to 1.11 immediately. Please
let us know if you could fix your problem, otherwise we can try to loop
in other people that should know better.

Regards,
Timo


[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-memory-off-heap



On 25.02.21 15:50, Bariša wrote:
> Small update:
>   we believe that the off heap memory is used by the parquet writer (
> used in sink to write to S3 )
>
> On Wed, 24 Feb 2021 at 23:25, Bariša <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I'm running flink 1.8.2 in a container, and under heavy load,
>     container gets OOM from the kernel.
>     I'm guessing that that reason for the kernel OOM is large size of
>     the off-heap memory. Is there a way I can limit it in flink 1.8.2?
>
>     I can see that newer version of flink has a config param, checking
>     here is it possible to do something similar in flink 1.8.2, without
>     a flink upgrade?
>
>     Cheers,
>     Barisa
>