java.lang.OutOfMemoryError: Java heap space at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:62)

[sohimankotia]

Hi,

I am getting Java Heap Space error while running Flink Job (Flink 1.2 ) .

Use case :  I am getting all keys from REDIS with specific pattern . Then
streaming over those keys and reading data from Redis for those key and
writing to file in HDFS .

Job was running fine for few days but suddenly started giving Heap Space
error .

Job Parallelism : 9 ( 3 nodes and 3 slots)

Total Redis Keys  : 1031966  ( aprox 33 MB in size (keys only))



Logs :







--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Stefan Richter]

Hi,

could you give us some more information like the size of your heap space, information about where and how you implemented access to Redis and how you kep the retrieved data, and most importantly a stack trace or (much better) a log?

Best,
Stefan

> Am 26.09.2017 um 06:52 schrieb sohimankotia <[hidden email]>:
>
> Hi,
>
> I am getting Java Heap Space error while running Flink Job (Flink 1.2 ) .
>
> Use case :  I am getting all keys from REDIS with specific pattern . Then
> streaming over those keys and reading data from Redis for those key and
> writing to file in HDFS .
>
> Job was running fine for few days but suddenly started giving Heap Space
> error .
>
> Job Parallelism : 9 ( 3 nodes and 3 slots)
>
> Total Redis Keys  : 1031966  ( aprox 33 MB in size (keys only))
>
>
>
> Logs :
>
>
>
>
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[sohimankotia]

Hi Stefan ,

Here is main class code  :

final String outFile = getOutFileName(backupDir);

final Set<String> keys = getAllRedisKeys(parameters);

final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

env.setParallelism(parallelism);

env

    .fromCollection(keys)

    .rebalance()

    .flatMap(new RedisRead()).withParameters(parameters.getConfiguration())

    .writeAsText(outFile);

env.execute();

RedisRead  -> simply read key one by one .

Memory Per task  : 2048 MB

Attaching flink logs .

Let me know if I can help anything more .

Thanks and Regards

Sohi

On Tue, Sep 26, 2017 at 2:50 PM, Stefan Richter <[hidden email]> wrote:

Hi,

could you give us some more information like the size of your heap space, information about where and how you implemented access to Redis and how you kep the retrieved data, and most importantly a stack trace or (much better) a log?

Best,
Stefan

> Am 26.09.2017 um 06:52 schrieb sohimankotia <[hidden email]>:
>
> Hi,
>
> I am getting Java Heap Space error while running Flink Job (Flink 1.2 ) .
>
> Use case :  I am getting all keys from REDIS with specific pattern . Then
> streaming over those keys and reading data from Redis for those key and
> writing to file in HDFS .
>
> Job was running fine for few days but suddenly started giving Heap Space
> error .
>
> Job Parallelism : 9 ( 3 nodes and 3 slots)
>
> Total Redis Keys  : 1031966  ( aprox 33 MB in size (keys only))
>
>
>
> Logs :
>
>
>
>
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[sohimankotia]

Hi Stefan ,

Here is main class code  :

final String outFile = getOutFileName(backupDir);

final Set<String> keys = getAllRedisKeys(parameters);

final ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(parallelism);

env
    .fromCollection(keys)
    .rebalance()
    .flatMap(new RedisRead()).withParameters(parameters.getConfiguration())
    .writeAsText(outFile);

env.execute();


RedisRead  -> simply read key one by one .


Memory Per task  : 2048 MB

Attaching flink logs .

Let me know if I can help anything more .
error.log
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t894/error.log>  






--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/