Configuring taskmanager with HA jobmanager

[John Sherwood]

I'm attempting to move to an HA configuration with a trio of JobManagers on top of a ZK cluster. From the docs, it appears that I should have them in my 'masters' file (as I do), but when I attempt to start the TaskManagers, they die complaining there is no jobmanager.rpc.address config - which seems counter to the purpose of having the masters. Is there some other setting I'm missing to tell the taskmanagers to use the masters file?

[Ufuk Celebi]

Hey John!

It looks like the task managers are not picking up the correct
configuration. Can you please verify that all nodes (JobManager and
TaskManager) use the same configuration.

The task managers use ZooKeeper to look up the JobManager and not the
configuration.


From the docs (https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/jobmanager_high_availability.html),
you need at least the following configuration:

recovery.mode: zookeeper
recovery.zookeeper.quorum: localhost:2181
recovery.zookeeper.path.root: /flink # important: customize per cluster
state.backend: filesystem
state.backend.fs.checkpointdir: hdfs:///flink/checkpoints
recovery.zookeeper.storageDir: hdfs:///flink/recovery


Does this help?

– Ufuk


On Wed, May 4, 2016 at 8:42 PM, John Sherwood <[hidden email]> wrote:
> I'm attempting to move to an HA configuration with a trio of JobManagers on
> top of a ZK cluster. From the docs, it appears that I should have them in my
> 'masters' file (as I do), but when I attempt to start the TaskManagers, they
> die complaining there is no jobmanager.rpc.address config - which seems
> counter to the purpose of having the masters. Is there some other setting
> I'm missing to tell the taskmanagers to use the masters file?