Fwd: Contact from site - Lou

[Kostas Tzoumas]

Hi,

I am forwarding this question from Lou so that others can benefit as well.

Kostas

---------- Forwarded message ----------
From: <[hidden email]>
Date: Thu, Mar 26, 2015 at 4:07 PM


Hi there,

I am Lou, a Ph.D. research scientist working at Ericsson Research (ER), Stockholm, Sweden.

At first, many thanks for the presentation given by Kostas and Stephan at ER this Tuesday, about which I attended via telephone conference.

Next, may I ask you a short question please? We are currently working with the development of an in-house cluster/job manager, and my question is: is it easy to test our own job manager on Apache Flink, and do you have any guidelines about doing it? Moreover, is the current standalone cluster manager called “direct” on Flink?

Thanks in advance,

Cheers,
Lou

[Maximilian Michels]

Hi Lou,

Thanks for your interest in Apache Flink.

What exactly do you mean by "cluster/job manager"? Are you referring to a cluster resource management environment like Hadoop YARN? Currently, Flink has a standalone and a YARN execution mode. In the YARN case, cluster resource allocation is handled by YARN. Flink instances are spawned on the assigned resources. It should be possible to integrate a different resource management into Flink although no effort has been dedicated so far to do this in a systematic way.

By the way, Flink's component which is in charge of the scheduling and monitoring Flink jobs is also called JobManager. Replacing this component should be non-trivial because it is hardwired with Flink's execution logic.

Best,

Max

On Fri, Mar 27, 2015 at 9:08 AM, Kostas Tzoumas <[hidden email]> wrote:

Hi,

I am forwarding this question from Lou so that others can benefit as well.

Kostas

---------- Forwarded message ----------
From: <[hidden email]>
Date: Thu, Mar 26, 2015 at 4:07 PM


Hi there,

I am Lou, a Ph.D. research scientist working at Ericsson Research (ER), Stockholm, Sweden.

At first, many thanks for the presentation given by Kostas and Stephan at ER this Tuesday, about which I attended via telephone conference.

Next, may I ask you a short question please? We are currently working with the development of an in-house cluster/job manager, and my question is: is it easy to test our own job manager on Apache Flink, and do you have any guidelines about doing it? Moreover, is the current standalone cluster manager called “direct” on Flink?

Thanks in advance,

Cheers,
Lou

[Maximilian Michels]

Hi Lou,

1. In the current Flink implementation, does TaskManager (on each worker) provide some information about monitoring the task queue status on the worker? Or we should actually develop such a functionality on our own.

The TaskManager does not provide information about the task queue status. All that information is hold by the JobManager. We recently integrated some monitoring features as of this pull request: https://github.com/apache/flink/pull/421 which help to monitor the resources at the task managers.

2. Can you provide me with a bit more information about how the resource management is taken care in the Flink standalone mode? Say, if there are three jobs submitted by users, Flink JobManager will dispatch such jobs in the FIFO manner, which will be further executed by a few workers under different scheduling policies defined by Flink TaskManager? I tried to collect such information online, which was unfortunately not satisfactory…

Currently, Flink does't support proper sharing of resources for multiple jobs. Multiple jobs can be deployed but it's scheduling might fail because of unavailable resources. So you have to make sure that you divide the resources among multiple jobs by planning beforehand. This will change very soon as we're currently working on changing this behavior.

3. One last question, i.e. by considering an example of users submitting two job requests in parallel, it is possible for the Flink system to have two clients to submit such requests to two JobManagers which will dispatch ExecutionGraph to a few TaskManagers concurrently? I think this case is a bit different comparing the ones as shown on the Apache Flink tutorial slides.

Absolutely. Two clients may submit a job at the same time. Just like pointed out above, scheduling of one job might fail because of unavailable resources. Whoever wins, depends on the timing. The JobManager's scheduler holds a lock which will ensure that concurrent scheduling of jobs does not result in a conflicting state (e.g. race condition).

I'll be glad to answer any other questions you might have. I think you are far from misconceiving Flink :)

Best regards,

Max

On Fri, Mar 27, 2015 at 11:16 PM, Yue Lu <[hidden email]> wrote:

Hi Max,

 

Thanks for the reply, and should you please see my comments below.

 > What exactly do you mean by "cluster/job manager"? Are you referring to a cluster resource management environment like Hadoop YARN? Currently, Flink has a standalone and a YARN execution mode.

 

After reading a bit more about Flink, I think it was referred to JobManager, which transforms the received JobGraph (pre-processed by the client) to ExecutionGraph, which will be further deployed and executed on workers by TaskManagers in Flink.

Next, I do have a few following questions in short, i.e.

1. In the current Flink implementation, does TaskManager (on each worker) provide some information about monitoring the task queue status on the worker? Or we should actually develop such a functionality on our own. 

2. Can you provide me with a bit more information about how the resource management is taken care in the Flink standalone mode? Say, if there are three jobs submitted by users, Flink JobManager will dispatch such jobs in the FIFO manner, which will be further executed by a few workers under different scheduling policies defined by Flink TaskManager? I tried to collect such information online, which was unfortunately not satisfactory…

 

3. One last question, i.e. by considering an example of users submitting two job requests in parallel, it is possible for the Flink system to have two clients to submit such requests to two JobManagers which will dispatch ExecutionGraph to a few TaskManagers concurrently? I think this case is a bit different comparing the ones as shown on the Apache Flink tutorial slides.

 

Last but not least, if there are some questions based off of my misconception about Flink, I am just so sorry about it. ;)

Thanks in advance, and have a nice weekend there.

 

Cheers,

Lou

 

Yue Lu, Ph.D. 

Research Scientist in Cloud Core and Architecture Research, RACT

Ericsson Research

Färögatan 6
164 80, Kista, Sweden
Mobile             <a href="tel:%2B46%280%29730958246" value="[phone]" target="_blank">[phone]

Fixed phone     <a href="tel:%2B46%280%29107158246" value="[phone]" target="_blank">[phone]

http://se.linkedin.com/pub/yue-lu/12/699/71

 

This communication is confidential. We only send and receive email on the basis of the terms set out at www.ericsson.com/email_disclaimer

 

---

From: Maximilian Michels [[hidden email]]
Sent: Friday, 27 March 2015 1:29 PM
To: [hidden email]
Cc: Yue Lu
Subject: Re: Contact from site - Lou

Hi Lou,

Thanks for your interest in Apache Flink.

What exactly do you mean by "cluster/job manager"? Are you referring to a cluster resource management environment like Hadoop YARN? Currently, Flink has a standalone and a YARN execution mode. In the YARN case, cluster resource allocation is handled by YARN. Flink instances are spawned on the assigned resources. It should be possible to integrate a different resource management into Flink although no effort has been dedicated so far to do this in a systematic way.

By the way, Flink's component which is in charge of the scheduling and monitoring Flink jobs is also called JobManager. Replacing this component should be non-trivial because it is hardwired with Flink's execution logic.

Best,

Max

On Fri, Mar 27, 2015 at 9:08 AM, Kostas Tzoumas <[hidden email]> wrote:

Hi,

I am forwarding this question from Lou so that others can benefit as well.

Kostas

---------- Forwarded message ----------
From: <[hidden email]>
Date: Thu, Mar 26, 2015 at 4:07 PM


Hi there,

I am Lou, a Ph.D. research scientist working at Ericsson Research (ER), Stockholm, Sweden.

At first, many thanks for the presentation given by Kostas and Stephan at ER this Tuesday, about which I attended via telephone conference.

Next, may I ask you a short question please? We are currently working with the development of an in-house cluster/job manager, and my question is: is it easy to test our own job manager on Apache Flink, and do you have any guidelines about doing it? Moreover, is the current standalone cluster manager called “direct” on Flink?

Thanks in advance,

Cheers,
Lou