Cluster Memory Error

[Will Walters]

Hello,

I am attempting to use the Flink runner to execute a data pipeline on a Hadoop cluster. The pipeline is created successfully in Flink, but when it attempts to execute the tasks, I receive a 'Size of total memory must be positive' error, despite having plenty of memory.

Here's the log around the area where the error occurs:

17/06/02 22:06:06 INFO taskmanager.TaskManager: Messages between TaskManager and JobManager have a max timeout of 10000 milliseconds

17/06/02 22:06:06 INFO taskmanager.TaskManager: Temporary file directory '/tmp': total 416 GB, usable 310 GB (74.52% usable)

17/06/02 22:06:06 INFO jobmanager.JobManager: JobManager akka://flink/user/jobmanager_1 was granted leadership with leader session ID None.

17/06/02 22:06:06 INFO buffer.NetworkBufferPool: Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768).

17/06/02 22:06:06 INFO standalone.StandaloneResourceManager: Resource Manager associating with leading JobManager Actor[akka://flink/user/jobmanager_1#2084519917] - leader session null

17/06/02 22:06:06 INFO network.NetworkEnvironment: Starting the network environment and its components.

17/06/02 22:06:06 INFO taskmanager.TaskManager: Limiting managed memory to 17592186044406 MB, memory will be allocated lazily.

17/06/02 22:06:06 ERROR flink.FlinkRunner: Pipeline execution failed

java.lang.IllegalArgumentException: Size of total memory must be positive.

at org.apache.flink.runtime.memory.MemoryManager.<init>(MemoryManager.java:131)

Any help would be much appreciated!

Thanks,

Will Walters.

[Aljoscha Krettek]

Hi Will,

You you please share some details regarding how you start your cluster and how you submit the program?

Best,

Aljoscha

On 3. Jun 2017, at 00:26, Will Walters <[hidden email]> wrote:

Hello,

I am attempting to use the Flink runner to execute a data pipeline on a Hadoop cluster. The pipeline is created successfully in Flink, but when it attempts to execute the tasks, I receive a 'Size of total memory must be positive' error, despite having plenty of memory.

Here's the log around the area where the error occurs:

17/06/02 22:06:06 INFO taskmanager.TaskManager: Messages between TaskManager and JobManager have a max timeout of 10000 milliseconds

17/06/02 22:06:06 INFO taskmanager.TaskManager: Temporary file directory '/tmp': total 416 GB, usable 310 GB (74.52% usable)

17/06/02 22:06:06 INFO jobmanager.JobManager: JobManager <a href="akka://flink/user/jobmanager_1" class="">akka://flink/user/jobmanager_1 was granted leadership with leader session ID None.

17/06/02 22:06:06 INFO buffer.NetworkBufferPool: Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768).

17/06/02 22:06:06 INFO standalone.StandaloneResourceManager: Resource Manager associating with leading JobManager Actor[<a href="akka://flink/user/jobmanager_1#2084519917" class="">akka://flink/user/jobmanager_1#2084519917] - leader session null

17/06/02 22:06:06 INFO network.NetworkEnvironment: Starting the network environment and its components.

17/06/02 22:06:06 INFO taskmanager.TaskManager: Limiting managed memory to 17592186044406 MB, memory will be allocated lazily.

17/06/02 22:06:06 ERROR flink.FlinkRunner: Pipeline execution failed

java.lang.IllegalArgumentException: Size of total memory must be positive.

at org.apache.flink.runtime.memory.MemoryManager.<init>(MemoryManager.java:131)

Any help would be much appreciated!

Thanks,

Will Walters.