Flink 1.7.1 - Parallel tasks in different Task Managers

[Francisco Blaya]

Hi,

We are trying to upgrade to Flink 1.7.1. We have a job with parallelism > 1 over 2 tasks. Due to implementation details outside our control one of these tasks cannot share the same JVM with another parallel instance i.e. if the parallelism is 4 we need 4 slots across different Task Managers.

That requirement was met in Flink 1.6.1. Upfront provisioning of Task Managers (8 slots each) and deploying that job first think into the cluster did the trick.

However in Flink 1.7.1 it looks like Task Managers are drawn on demand as you deploy jobs. As a consequence we cannot get the parallel tasks deployed to different Task Managers.

We have thought of having a separate cluster just for that job where we can configure the Task Managers with one slot each. This way we'll get one parallel instance per Task Manager although we haven't tried it out.

Can anyone think of a way of making that work without requiring a separate cluster for that single job?

Regards,

Fran

hivehome.com

Hive | London | Cambridge | Houston | Toronto

The information contained in or attached to this email is confidential and intended only for the use of the individual(s) to which it is addressed. It may contain information which is confidential and/or covered by legal professional or other privilege. The views expressed in this email are not necessarily the views of Centrica plc, and the company, its directors, officers or employees make no representation or accept any liability for their accuracy or completeness unless expressly stated to the contrary. 

Centrica Hive Limited (company no: 5782908), registered in England and Wales with its registered office at Millstream, Maidenhead Road, Windsor, Berkshire SL4 5GD.