Raspberry Pi Memory Configuration

[Nicholas Walton]

Hi,

Hope this is the right place to ask, but I have a 5 Raspberry Pi 3 cluster running Flink on which I’m hitting memory issues. Each Pi has 1Gb and is running with a 256Gb USB drive, however I don’t seem to be able to configure Java to use more than 256M of heap. The memory problem, I’m reasonably sure, lies in in the key store backed by the filestore (state.backend: filesystem). Either it is not caching to disk or has insufficient RAM to function properly. How can I set the configuration to increase the heap size?

The task manger pane shows the following for each of the five PIs

<a href="akka.tcp://flink@wright:35869/user/taskmanager" class="">akka.tcp://flink@wright:35869/user/taskmanager 456052018-04-28, 10:35:25224923 MB224 MB108 MB

My flink configuration is 

#—————————————————————————————————

jobmanager.rpc.address: gandalf

# The RPC port where the JobManager is reachable.

jobmanager.rpc.port: 6123

# The heap size for the JobManager JVM, I have to comment these out 

# otherwise Flink refuses to run on the Pi

# jobmanager.heap.mb: 1024

# The heap size for the TaskManager JVM

# taskmanager.heap.mb: 1024

# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.

taskmanager.numberOfTaskSlots: 2

# Specify whether TaskManager memory should be allocated when starting up (true) or when

# memory is required in the memory manager (false)

# Important Note: For pure streaming setups, we highly recommend to set this value to `false`

# as the default state backends currently do not use the managed memory.

taskmanager.memory.preallocate: false

# The parallelism used for programs that did not specify and other parallelism.

parallelism.default: 2

….

state.backend: filesystem

….

#—————————————————————————————————

[Kien Truong]

Hi Nicholas,

Try reducing

containerized.heap-cutoff-min

from the default 600MB to a lower number.

Still, I think RPi is way too under-powered to run any serious Flink apps.

Regards,

Kien

On 4/28/2018 4:38 PM, Nicholas Walton wrote:

Hi,

Hope this is the right place to ask, but I have a 5 Raspberry Pi 3 cluster running Flink on which I’m hitting memory issues. Each Pi has 1Gb and is running with a 256Gb USB drive, however I don’t seem to be able to configure Java to use more than 256M of heap. The memory problem, I’m reasonably sure, lies in in the key store backed by the filestore (state.backend: filesystem). Either it is not caching to disk or has insufficient RAM to function properly. How can I set the configuration to increase the heap size?

The task manger pane shows the following for each of the five PIs

<a href="akka.tcp://flink@wright:35869/user/taskmanager" class="" moz-do-not-send="true">akka.tcp://flink@wright:35869/user/taskmanager 456052018-04-28, 10:35:25224923 MB224 MB108 MB

My flink configuration is 

#—————————————————————————————————

jobmanager.rpc.address: gandalf

# The RPC port where the JobManager is reachable.

jobmanager.rpc.port: 6123

# The heap size for the JobManager JVM, I have to comment these out 

# otherwise Flink refuses to run on the Pi

# jobmanager.heap.mb: 1024

# The heap size for the TaskManager JVM

# taskmanager.heap.mb: 1024

# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.

taskmanager.numberOfTaskSlots: 2

# Specify whether TaskManager memory should be allocated when starting up (true) or when

# memory is required in the memory manager (false)

# Important Note: For pure streaming setups, we highly recommend to set this value to `false`

# as the default state backends currently do not use the managed memory.

taskmanager.memory.preallocate: false

# The parallelism used for programs that did not specify and other parallelism.

parallelism.default: 2

….

state.backend: filesystem

….

#—————————————————————————————————

[Nicholas Walton]

Thanks for the reply, but I tracked the problem down to a missing M in task m=
anager.sh

# export JVM_ARGS=3D"${JVM_ARGS} -Xms${TM_HEAP_SIZE}M -Xmx${TM_HEAP_SIZE}M -=
XX:MaxDirectMemorySize=3D${TM_MAX_OFFHEAP_SIZE}"
export JVM_ARGS=3D"${JVM_ARGS} -Xms${TM_HEAP_SIZE}M -Xmx${TM_HEAP_SIZE}M"

I had to remove {TM_MAX_OFFHEAP_SIZE} which caused the Pi Java to barf, and h=
ad accidentally deleted the final M.

After that I could run with a heap set to 512m with minimal problem.

As for the Pi as a Flink client, it may not be powerful in conventional sens=
e but I can directly attach a cluster to a sensor array and process the data=
in real time from multiple streams, which is neat. I processed 3 million re=
adings in the prototype in just over an hour with a modest amount of process=
ing required, all on 5pis with a sixth acting as the job manager. My only co=
mplaint is that Flink has a habit of dropping task managers even though they=
're still running. I suspect that's because of slow Ethernet on the Pi3 whic=
h has been addressed on the Pi3+ just released. Realistically sized rack mou=
nted clusters are now possible from http://www.bitscope.com/product/blade/ a=
fter Los Alamos National Lab went looking for a cheap means to build prototy=
pe clusters, and settled on the Pi to build a 750+ node cluster.

[Ankit Chaudhary]

Hey Nicholas,

I am not sure if the problem of Flink dropping TM is connected to ethernet because it happens in our Prod cluster too where we have standard blade servers with Gigabyte network (din spent time yet on doing the RCA).

Nonetheless, its really cool what you are doing there with rPis :))). 

Cheers, Ankit 

On Sun, Apr 29, 2018 at 4:45 PM, Nicholas Walton <[hidden email]> wrote:

Thanks for the reply, but I tracked the problem down to a missing M in task m=
anager.sh

# export JVM_ARGS=3D"${JVM_ARGS} -Xms${TM_HEAP_SIZE}M -Xmx${TM_HEAP_SIZE}M -=
XX:MaxDirectMemorySize=3D${TM_MAX_OFFHEAP_SIZE}"
export JVM_ARGS=3D"${JVM_ARGS} -Xms${TM_HEAP_SIZE}M -Xmx${TM_HEAP_SIZE}M"

I had to remove {TM_MAX_OFFHEAP_SIZE} which caused the Pi Java to barf, and h=
ad accidentally deleted the final M.

After that I could run with a heap set to 512m with minimal problem.

As for the Pi as a Flink client, it may not be powerful in conventional sens=
e but I can directly attach a cluster to a sensor array and process the data=
in real time from multiple streams, which is neat. I processed 3 million re=
adings in the prototype in just over an hour with a modest amount of process=
ing required, all on 5pis with a sixth acting as the job manager. My only co=
mplaint is that Flink has a habit of dropping task managers even though they=
're still running. I suspect that's because of slow Ethernet on the Pi3 whic=
h has been addressed on the Pi3+ just released. Realistically sized rack mou=
nted clusters are now possible from http://www.bitscope.com/product/blade/ a=
fter Los Alamos National Lab went looking for a cheap means to build prototy=
pe clusters, and settled on the Pi to build a 750+ node cluster.