Configuring hdfs with the MesosApplicationMasterRunner

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Configuring hdfs with the MesosApplicationMasterRunner

Guillaume Balaine
Hi,

I get that this might not be a priority given FLIP6, but I set up a mesos configuration for clustering with Flink today, and one of the painful points was the hadoop configuration.

It's impossible to pass the hadoop conf folder artefact (containing files like core-site.xml etc.) to the task managers. The appmaster clearly sees the hadoop configuration folder but then even if I setup container volumes for the task managers, I can't mount these files.

I read the whole flink-mesos code but there was no way to pass the hdfs file. 
I see that flink-dcos-service has a FLINK_HDFS_CONFIG_URL env var but it does nothing... 
Which means I have to either put them directly in the container images (what I did), or download them at startup (inconvenient).

Has anyone come across this kind of ops problem before ?