Running on a firewalled Yarn cluster?

Posted by Niels Basjes on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Running-on-a-firewalled-Yarn-cluster-tp3330.html

Hi,

Here at work our security guys have chosen (long time ago) to only allow the firewalls to have the ports open that needed (I say: good call!).

For the Yarn cluster this includes things like the proxy to see the application manager of an application.

For everything we've done so far (i.e. mr/pig/...) this has worked fine.

Now with Flink I run into problems: 

When I run either the yarn-session or a job on Yarn the application manager gets started and I can see the webinterface.

The problem is that the jobmanager.rpc.address is on one of the worker nodes and the jobmanager.rpc.port is essentially a random value.

A random value which is not accessible because of the firewall rules.

So I cannot reach the jobmanager on the yarn cluster.

How do I tackle this assuming that opening the all ports on the firewall is not an option?

Or is this something that should be handled by Flink? ( Perhaps the application manager can proxy the RPC calls? )

--