Running the JobManager and TaskManager on the same node in a cluster

[Dominik Safaric]

Hi,

It is generally recommended for streaming engines, also including Flink to run a separate master node - in the case of Flink, the JobManager.

However, why should one in Flink run the JobManager on a separate node?

Performance wise, the JobManager isn’t intense unlike of course TaskManagers.

In terms of fault tolerance and a failing JobManager, semantically there is no difference.

Hence, what are the main reasons behind this rationale?

Thanks in advance.  

[rmetzger0]

Hi Dominik,

Your observation is right, running the JobManager and TaskManager on the same node is no problem. If that machine fails, both services will be affected, but as long as you have infrastructure in place (YARN for example) to start them somewhere else, nothing bad will happen.

Regarding your initial statement, that "It is generally recommended for streaming engines, also including Flink to run a separate master node", did you read that somewhere in the Flink documentation?

Regards,

Robert

On Wed, Nov 16, 2016 at 9:49 PM, Dominik Safaric <[hidden email]> wrote:

Hi,

It is generally recommended for streaming engines, also including Flink to run a separate master node - in the case of Flink, the JobManager.

However, why should one in Flink run the JobManager on a separate node?

Performance wise, the JobManager isn’t intense unlike of course TaskManagers.

In terms of fault tolerance and a failing JobManager, semantically there is no difference.

Hence, what are the main reasons behind this rationale?

Thanks in advance.