Isolate Tasks - Run Distinct Tasks in Different Task Managers

[PedroMrChaves]

Hello,

Assuming that I have the following Job Graph,
       
(Source) -> (map) -> (KeyBy | Window | apply) -> (Sink)

Is there a way to assure that the map operator (and all its subtasks) run on a different
task manager than the operator (map | window | apply)?

This would allow JVM memory isolation without using YARN.

Regards,
Pedro

[Ufuk Celebi]

Internally, Flink defines through SlotSharingGroup which tasks may
share a task manager slot. By configuring each TaskManager to have a
single slot and configuring the slot sharing groups accordingly, you
can get the desired behaviour.

You can specify the slot sharing group for an operator like map by
calling slotSharingGroup(String). You would have you set a different
slot sharing group for the window/apply part.

Does this help?


On Wed, Mar 8, 2017 at 10:44 AM, PedroMrChaves
<[hidden email]> wrote:

> Hello,
>
> Assuming that I have the following Job Graph,
>
> (Source) -> (map) -> (KeyBy | Window | apply) -> (Sink)
>
> Is there a way to assure that the map operator (and all its subtasks) run on
> a different
> task manager than the operator (map | window | apply)?
>
> This would allow JVM memory isolation without using YARN.
>
> Regards,
> Pedro
>
>
>
>
>
> -----
> Best Regards,
> Pedro Chaves
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Isolate-Tasks-Run-Distinct-Tasks-in-Different-Task-Managers-tp12104.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[PedroMrChaves]

Thanks for the response.

I would like to assure that the map operator is not in the same task manager as the window/apply operator, regardless of the number of slots of each task manager.

[rmetzger0]

There is currently no way in Flink to define such scheduling constraints.

On Wed, Mar 8, 2017 at 5:00 PM, PedroMrChaves <[hidden email]> wrote:

Thanks for the response.

I would like to assure that the map operator is not in the same task manager
as the window/apply operator, regardless of the number of slots of each task
manager.



-----
Best Regards,
Pedro Chaves
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Isolate-Tasks-Run-Distinct-Tasks-in-Different-Task-Managers-tp12104p12118.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[PedroMrChaves]

Can YARN provide task isolation?

[Stephan Ewen]

Yes, simply set the number of slots per TaskManager in YARN to 1. That gives you the isolation.

On Tue, Mar 14, 2017 at 11:58 AM, PedroMrChaves <[hidden email]> wrote:

Can YARN provide task isolation?




-----
Best Regards,
Pedro Chaves
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Isolate-Tasks-Run-Distinct-Tasks-in-Different-Task-Managers-tp12104p12201.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.