Re: Raspberry Pi Memory Configuration

Posted by Ankit Chaudhary on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Raspberry-Pi-Memory-Configuration-tp19848p19862.html

Hey Nicholas,

I am not sure if the problem of Flink dropping TM is connected to ethernet because it happens in our Prod cluster too where we have standard blade servers with Gigabyte network (din spent time yet on doing the RCA).

Nonetheless, its really cool what you are doing there with rPis :))). 

Cheers, Ankit 

On Sun, Apr 29, 2018 at 4:45 PM, Nicholas Walton <[hidden email]> wrote:

Thanks for the reply, but I tracked the problem down to a missing M in task m=
anager.sh

# export JVM_ARGS=3D"${JVM_ARGS} -Xms${TM_HEAP_SIZE}M -Xmx${TM_HEAP_SIZE}M -=
XX:MaxDirectMemorySize=3D${TM_MAX_OFFHEAP_SIZE}"
export JVM_ARGS=3D"${JVM_ARGS} -Xms${TM_HEAP_SIZE}M -Xmx${TM_HEAP_SIZE}M"

I had to remove {TM_MAX_OFFHEAP_SIZE} which caused the Pi Java to barf, and h=
ad accidentally deleted the final M.

After that I could run with a heap set to 512m with minimal problem.

As for the Pi as a Flink client, it may not be powerful in conventional sens=
e but I can directly attach a cluster to a sensor array and process the data=
in real time from multiple streams, which is neat. I processed 3 million re=
adings in the prototype in just over an hour with a modest amount of process=
ing required, all on 5pis with a sixth acting as the job manager. My only co=
mplaint is that Flink has a habit of dropping task managers even though they=
're still running. I suspect that's because of slow Ethernet on the Pi3 whic=
h has been addressed on the Pi3+ just released. Realistically sized rack mou=
nted clusters are now possible from http://www.bitscope.com/product/blade/ a=
fter Los Alamos National Lab went looking for a cheap means to build prototy=
pe clusters, and settled on the Pi to build a 750+ node cluster.