Multiple Task Slots support in Flink 1.5

[Abdul Qadeer]

Hi!

I came across the following point in release notes of 1.5 version:

"The allocation of TaskManagers with multiple slots is not fully supported yet."

Does this mean the support for it will come as a patch for 1.5? or will it be in the next stable release?

If I use legacy mode, will that support multiple slots per TaskManager?, or is it only the deployment change that will get affected?

[Kien Truong]

Hi,

We're using multiple slots per TaskManager with legacy mode, and everything works fine.

For the new default mode, it also seems to works for us, so I'm not sure what is not supported. May be someone from Flink team could clarify.

Best regards,

Kien

On 5/31/2018 4:26 AM, Abdul Qadeer wrote:

Hi!

I came across the following point in release notes of 1.5 version:

"The allocation of TaskManagers with multiple slots is not fully supported yet."

Does this mean the support for it will come as a patch for 1.5? or will it be in the next stable release?

If I use legacy mode, will that support multiple slots per TaskManager?, or is it only the deployment change that will get affected?

[Fabian Hueske-2]

Hi,

The release notes state that "multiple slots are not *fully* supported".

In Flink 1.5.0, the configured number of slots is ignored when requesting containers for TaskManagers from a resource manager, i.e., Flink assumes TMs with 1 slot.

Hence, Flink request too many containers and starts too many TMs, but each TM is started with the correct number of slots.
All unused containers are returned after a configurable time out.

The problem can be prevented by configuring 1 slot per TM.

Best, Fabian

2018-05-31 14:12 GMT+02:00 Kien Truong <[hidden email]>:

Hi,

We're using multiple slots per TaskManager with legacy mode, and everything works fine.

For the new default mode, it also seems to works for us, so I'm not sure what is not supported. May be someone from Flink team could clarify.

Best regards,

Kien

On 5/31/2018 4:26 AM, Abdul Qadeer wrote:

Hi!

I came across the following point in release notes of 1.5 version:

"The allocation of TaskManagers with multiple slots is not fully supported yet."

Does this mean the support for it will come as a patch for 1.5? or will it be in the next stable release?

If I use legacy mode, will that support multiple slots per TaskManager?, or is it only the deployment change that will get affected?

[Abdul Qadeer]

Hi Kien!

Thank you for sharing your experience. Do you have Flink deployed in Standalone Cluster mode or K8S/Docker mode?

On Fri, 1 Jun 2018 at 00:55, Fabian Hueske <[hidden email]> wrote:

Hi,

The release notes state that "multiple slots are not *fully* supported".

In Flink 1.5.0, the configured number of slots is ignored when requesting containers for TaskManagers from a resource manager, i.e., Flink assumes TMs with 1 slot.

Hence, Flink request too many containers and starts too many TMs, but each TM is started with the correct number of slots.
All unused containers are returned after a configurable time out.

The problem can be prevented by configuring 1 slot per TM.

Best, Fabian

2018-05-31 14:12 GMT+02:00 Kien Truong <[hidden email]>:

Hi,

We're using multiple slots per TaskManager with legacy mode, and everything works fine.

For the new default mode, it also seems to works for us, so I'm not sure what is not supported. May be someone from Flink team could clarify.

Best regards,

Kien

On 5/31/2018 4:26 AM, Abdul Qadeer wrote:

Hi!

I came across the following point in release notes of 1.5 version:

"The allocation of TaskManagers with multiple slots is not fully supported yet."

Does this mean the support for it will come as a patch for 1.5? or will it be in the next stable release?

If I use legacy mode, will that support multiple slots per TaskManager?, or is it only the deployment change that will get affected?

[Till Rohrmann]

FYI: https://issues.apache.org/jira/browse/FLINK-9455 tracks the problem with the multi slot support for the SlotManager.

Cheers,

Till

On Fri, Jun 1, 2018 at 10:10 AM, Abdul Qadeer <[hidden email]> wrote:

Hi Kien!

Thank you for sharing your experience. Do you have Flink deployed in Standalone Cluster mode or K8S/Docker mode?

On Fri, 1 Jun 2018 at 00:55, Fabian Hueske <[hidden email]> wrote:

Hi,

The release notes state that "multiple slots are not *fully* supported".

In Flink 1.5.0, the configured number of slots is ignored when requesting containers for TaskManagers from a resource manager, i.e., Flink assumes TMs with 1 slot.

Hence, Flink request too many containers and starts too many TMs, but each TM is started with the correct number of slots.
All unused containers are returned after a configurable time out.

The problem can be prevented by configuring 1 slot per TM.

Best, Fabian

2018-05-31 14:12 GMT+02:00 Kien Truong <[hidden email]>:

Hi,

We're using multiple slots per TaskManager with legacy mode, and everything works fine.

For the new default mode, it also seems to works for us, so I'm not sure what is not supported. May be someone from Flink team could clarify.

Best regards,

Kien

On 5/31/2018 4:26 AM, Abdul Qadeer wrote:

Hi!

I came across the following point in release notes of 1.5 version:

"The allocation of TaskManagers with multiple slots is not fully supported yet."

Does this mean the support for it will come as a patch for 1.5? or will it be in the next stable release?

If I use legacy mode, will that support multiple slots per TaskManager?, or is it only the deployment change that will get affected?

[Abdul Qadeer]

Thank you for creating this Till, so as I understand this won't be supported (for standalone cluster mode) even if legacy mode is enabled?

On Fri, Jun 1, 2018 at 6:21 AM, Till Rohrmann <[hidden email]> wrote:

FYI: https://issues.apache.org/jira/browse/FLINK-9455 tracks the problem with the multi slot support for the SlotManager.

Cheers,

Till

On Fri, Jun 1, 2018 at 10:10 AM, Abdul Qadeer <[hidden email]> wrote:

Hi Kien!

Thank you for sharing your experience. Do you have Flink deployed in Standalone Cluster mode or K8S/Docker mode?

On Fri, 1 Jun 2018 at 00:55, Fabian Hueske <[hidden email]> wrote:

Hi,

The release notes state that "multiple slots are not *fully* supported".

In Flink 1.5.0, the configured number of slots is ignored when requesting containers for TaskManagers from a resource manager, i.e., Flink assumes TMs with 1 slot.

Hence, Flink request too many containers and starts too many TMs, but each TM is started with the correct number of slots.
All unused containers are returned after a configurable time out.

The problem can be prevented by configuring 1 slot per TM.

Best, Fabian

2018-05-31 14:12 GMT+02:00 Kien Truong <[hidden email]>:

Hi,

We're using multiple slots per TaskManager with legacy mode, and everything works fine.

For the new default mode, it also seems to works for us, so I'm not sure what is not supported. May be someone from Flink team could clarify.

Best regards,

Kien

On 5/31/2018 4:26 AM, Abdul Qadeer wrote:

Hi!

I came across the following point in release notes of 1.5 version:

"The allocation of TaskManagers with multiple slots is not fully supported yet."

Does this mean the support for it will come as a patch for 1.5? or will it be in the next stable release?

If I use legacy mode, will that support multiple slots per TaskManager?, or is it only the deployment change that will get affected?

[Till Rohrmann]

For standalone mode it works for the legacy and the new mode. Only when using Yarn or Mesos, it can happen that you temporarily allocate too many container. The unused container will, however, be freed if they idle too long.

Cheers,

Till

On Fri, Jun 1, 2018, 21:52 Abdul Qadeer <[hidden email]> wrote:

Thank you for creating this Till, so as I understand this won't be supported (for standalone cluster mode) even if legacy mode is enabled?

On Fri, Jun 1, 2018 at 6:21 AM, Till Rohrmann <[hidden email]> wrote:

FYI: https://issues.apache.org/jira/browse/FLINK-9455 tracks the problem with the multi slot support for the SlotManager.

Cheers,

Till

On Fri, Jun 1, 2018 at 10:10 AM, Abdul Qadeer <[hidden email]> wrote:

Hi Kien!

Thank you for sharing your experience. Do you have Flink deployed in Standalone Cluster mode or K8S/Docker mode?

On Fri, 1 Jun 2018 at 00:55, Fabian Hueske <[hidden email]> wrote:

Hi,

The release notes state that "multiple slots are not *fully* supported".

In Flink 1.5.0, the configured number of slots is ignored when requesting containers for TaskManagers from a resource manager, i.e., Flink assumes TMs with 1 slot.

Hence, Flink request too many containers and starts too many TMs, but each TM is started with the correct number of slots.
All unused containers are returned after a configurable time out.

The problem can be prevented by configuring 1 slot per TM.

Best, Fabian

2018-05-31 14:12 GMT+02:00 Kien Truong <[hidden email]>:

Hi,

We're using multiple slots per TaskManager with legacy mode, and everything works fine.

For the new default mode, it also seems to works for us, so I'm not sure what is not supported. May be someone from Flink team could clarify.

Best regards,

Kien

On 5/31/2018 4:26 AM, Abdul Qadeer wrote:

Hi!

I came across the following point in release notes of 1.5 version:

"The allocation of TaskManagers with multiple slots is not fully supported yet."

Does this mean the support for it will come as a patch for 1.5? or will it be in the next stable release?

If I use legacy mode, will that support multiple slots per TaskManager?, or is it only the deployment change that will get affected?