Flink Queue Scheduling (JobManager)

[Vikram Saxena]

Hi 

I am reading and learning about Flink and I have tried to implement some Flink Jobs. 

In my application I have 2 Flink Jobs which I want to run in parallel.

Of course, as I understand I can have the task slots divided so that each one can run concurrently. 

But, is there a possibility for scheduling jobs only to be processed when the JobManager has resources free(task slots) and not throw back (NoResourceAvailableException) ??

The reason I ask this is because I have 2 Flink Jobs, one which would be always running in short intervals and other which would be in ad hoc basis. 

I do not want to have my task slots divided just for this ad hoc job, 

So I wanted to submit this adhoc job and the Job Manager can schedule it whenever it has free task slots ? 

Cheers,

Vikram

[Flavio Pompermaier]

That would be definitely interesting but I think that at the moment the only way to achieve that is to exploit YARN for that..

An integration with some job-workflow engine (like Apache Oozie and Apache Falcon) would also be very useful! I tried to wrote on Apache Oozie mailing list if there is any interest in integrating with apache Flink but I didn't receive any feedback yet :(

Best,

Flavio

On Wed, Apr 27, 2016 at 1:00 PM, Vikram Saxena <[hidden email]> wrote:

Hi 

I am reading and learning about Flink and I have tried to implement some Flink Jobs. 

In my application I have 2 Flink Jobs which I want to run in parallel.

Of course, as I understand I can have the task slots divided so that each one can run concurrently. 

But, is there a possibility for scheduling jobs only to be processed when the JobManager has resources free(task slots) and not throw back (NoResourceAvailableException) ??

The reason I ask this is because I have 2 Flink Jobs, one which would be always running in short intervals and other which would be in ad hoc basis. 

I do not want to have my task slots divided just for this ad hoc job, 

So I wanted to submit this adhoc job and the Job Manager can schedule it whenever it has free task slots ? 

Cheers,

Vikram

[Vikram Saxena]

Yes, I have tried using separate Yarn queues for this, but I have my doubts. 

Here is what I am trying to do:

I have 2 Flink Jobs 

JobA : Regular Job running every x minutes. 

JobB : User requested adhoc job 

Tried1.

Sol A : 

Have 2 queues on Yarn with 95 : 5 resource distribution 

Sol B: 

Have parallelism control the concurrent execution i.e. divide total task slots in 95:5 ratio 

In both the above solutions I would be dedicating a chunk of resources to  Job B.

I want Job A to run will all my resources and only if Job B is submitted, it should get some resources to process and not throw back NoResourceException. 

I hope I am able to explain my problem :) 

VG,

VIkram

On Wed, Apr 27, 2016 at 2:13 PM, Flavio Pompermaier <[hidden email]> wrote:

That would be definitely interesting but I think that at the moment the only way to achieve that is to exploit YARN for that..

An integration with some job-workflow engine (like Apache Oozie and Apache Falcon) would also be very useful! I tried to wrote on Apache Oozie mailing list if there is any interest in integrating with apache Flink but I didn't receive any feedback yet :(

Best,

Flavio

On Wed, Apr 27, 2016 at 1:00 PM, Vikram Saxena <[hidden email]> wrote:

Hi 

I am reading and learning about Flink and I have tried to implement some Flink Jobs. 

In my application I have 2 Flink Jobs which I want to run in parallel.

Of course, as I understand I can have the task slots divided so that each one can run concurrently. 

But, is there a possibility for scheduling jobs only to be processed when the JobManager has resources free(task slots) and not throw back (NoResourceAvailableException) ??

The reason I ask this is because I have 2 Flink Jobs, one which would be always running in short intervals and other which would be in ad hoc basis. 

I do not want to have my task slots divided just for this ad hoc job, 

So I wanted to submit this adhoc job and the Job Manager can schedule it whenever it has free task slots ? 

Cheers,

Vikram