Resource isolation in flink among multiple jobs

[Abhishek Agarwal]

In a standalone flink cluster, the TaskManager has multiple slots and can run different applications in different slot but still within same JVM. Given that, two applications are running under same process, how is the cpu and memory isolation achieved? 

Are there any recommendations to achieve isolation between applications? 

--

[rmetzger0]

Hi,

for isolation, we recommend using YARN, or soon Mesos.

For standalone installations, you'll need to manually set up multiple independent Flink clusters within one physical cluster if you want them to be isolated.

Regards,

Robert

On Mon, Aug 29, 2016 at 1:41 PM, Abhishek Agarwal <[hidden email]> wrote:

In a standalone flink cluster, the TaskManager has multiple slots and can run different applications in different slot but still within same JVM. Given that, two applications are running under same process, how is the cpu and memory isolation achieved? 

Are there any recommendations to achieve isolation between applications? 

--

[Abhishek Agarwal]

Hi Robert,

Thanks for your reply. Few follow up questions - 

Is there any timeline for mesos support? 

In standalone installations, how about having one Task slot per TaskManager and multiple TaskManager instances per machine? 

On Mon, Aug 29, 2016 at 7:13 PM, Robert Metzger <[hidden email]> wrote:

Hi,

for isolation, we recommend using YARN, or soon Mesos.

For standalone installations, you'll need to manually set up multiple independent Flink clusters within one physical cluster if you want them to be isolated.

Regards,

Robert

On Mon, Aug 29, 2016 at 1:41 PM, Abhishek Agarwal <[hidden email]> wrote:

In a standalone flink cluster, the TaskManager has multiple slots and can run different applications in different slot but still within same JVM. Given that, two applications are running under same process, how is the cpu and memory isolation achieved? 

Are there any recommendations to achieve isolation between applications? 

--

--

[rmetzger0]

Hi,

this is the JIRA for mesos support: https://issues.apache.org/jira/browse/FLINK-1984

The first part of it has been merged today: https://github.com/apache/flink/pull/2315

So its likely going to be fully functional for the 1.2 release.

You can run multiple TaskManager instances per machine, and give each only one (or very few) slots.

On Mon, Aug 29, 2016 at 5:40 PM, Abhishek Agarwal <[hidden email]> wrote:

Hi Robert,

Thanks for your reply. Few follow up questions - 

Is there any timeline for mesos support? 

In standalone installations, how about having one Task slot per TaskManager and multiple TaskManager instances per machine? 

On Mon, Aug 29, 2016 at 7:13 PM, Robert Metzger <[hidden email]> wrote:

Hi,

for isolation, we recommend using YARN, or soon Mesos.

For standalone installations, you'll need to manually set up multiple independent Flink clusters within one physical cluster if you want them to be isolated.

Regards,

Robert

On Mon, Aug 29, 2016 at 1:41 PM, Abhishek Agarwal <[hidden email]> wrote:

In a standalone flink cluster, the TaskManager has multiple slots and can run different applications in different slot but still within same JVM. Given that, two applications are running under same process, how is the cpu and memory isolation achieved? 

Are there any recommendations to achieve isolation between applications? 

--

--