Re: Use jvm to run flink on single-node machine with many cores

Posted by rmetzger0 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Use-jvm-to-run-flink-on-single-node-machine-with-many-cores-tp5062p5063.html

Hi Ana,

you can create a StreamExecutionEnvironment also by passing a configuration object. In the configuration, you can also configure the number of network buffers.

// set up the execution environment
Configuration conf = new Configuration();
conf.setBoolean("taskmanager.network.numberOfBuffers", "16000");
final StreamExecutionEnvironment env = LocalStreamEnvironment.createLocalEnvironment(8, conf);

On Sun, Feb 21, 2016 at 1:34 PM, Ana M. Martinez <[hidden email]> wrote:

Hi all,

I am trying to run a program using the flink java library with ExecutionEnvironment.getExecutionEnvironment() from the command line using java -jar.

If I run the code in my machine (with four cores) or in a multi-node cluster (using yarn) the program runs normally, but if I want to run it on a machine with a single node and 32 cores using java -jar I get the following error:

02/21/2016 13:33:09 MapPartition (MapPartition at toBatches(ConversionToBatches.java:55))(29/32) switched to FAILED 

java.io.IOException: Insufficient number of network buffers: required 1, but only 0 available. The total number of network buffers is currently set to 2048. You can increase this number by setting the configuration key 'taskmanager.network.numberOfBuffers'.

at org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.createBufferPool(NetworkBufferPool.java:196)

at org.apache.flink.runtime.io.network.NetworkEnvironment.registerTask(NetworkEnvironment.java:325)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:488)

at java.lang.Thread.run(Thread.java:745)

In this case (java -jar), I don’t know if or how I can increase the number of network buffers. Is there a way to do it without having to use yarn (as I don’t have hadoop installed)?

Thanks,

Ana