Flink with Yarn

[Sourav Mazumder]

I am going through the documentation of integrating Flink with YARN.

However not sure whether Flink can be run on YARN in two modes (like Spark). In one mode the driver/client program of Flink is also managed by YARN. In the second mode where the client program is outside the control of YARN. Is the running Flinkon behind Firewalls is like the second mode

Any clarification on this ?

Regards,

Sourav

[Stephan Ewen]

Hi!

Flink is different than Spark in that respect. The driver program in Flink can submit a program to the master (in YARN Application Master) and disconnect then. It is not a part of the distributed execution - that is coordinated only by the master (JobManager).

The driver can stay connected to receive progress updates, though.

For programs that do consist of multiple parallel executions (that have count() or collect() statements), the driver needs to stay connected, because it needs to pull the intermediate results. However, they are all pulled/proxied through the master (JobManager), so the driver needs not be able to connect to the workers. The only requirement for firewalled clusters is to have two ports from the master node reachable by the client.

Greetings,

Stephan

On Mon, Jan 11, 2016 at 5:18 PM, Sourav Mazumder <[hidden email]> wrote:

I am going through the documentation of integrating Flink with YARN.

However not sure whether Flink can be run on YARN in two modes (like Spark). In one mode the driver/client program of Flink is also managed by YARN. In the second mode where the client program is outside the control of YARN. Is the running Flinkon behind Firewalls is like the second mode

Any clarification on this ?

Regards,

Sourav

[Sourav Mazumder]

Hi Stephan,

Thanks for the explanation.

From your explanation it looks like Flink runs in a mode similar to Spark's YARN-Client mode.

Regards,

Sourav

On Mon, Jan 11, 2016 at 8:27 AM, Stephan Ewen <[hidden email]> wrote:

Hi!

Flink is different than Spark in that respect. The driver program in Flink can submit a program to the master (in YARN Application Master) and disconnect then. It is not a part of the distributed execution - that is coordinated only by the master (JobManager).

The driver can stay connected to receive progress updates, though.

For programs that do consist of multiple parallel executions (that have count() or collect() statements), the driver needs to stay connected, because it needs to pull the intermediate results. However, they are all pulled/proxied through the master (JobManager), so the driver needs not be able to connect to the workers. The only requirement for firewalled clusters is to have two ports from the master node reachable by the client.

Greetings,

Stephan

On Mon, Jan 11, 2016 at 5:18 PM, Sourav Mazumder <[hidden email]> wrote:

I am going through the documentation of integrating Flink with YARN.

However not sure whether Flink can be run on YARN in two modes (like Spark). In one mode the driver/client program of Flink is also managed by YARN. In the second mode where the client program is outside the control of YARN. Is the running Flinkon behind Firewalls is like the second mode

Any clarification on this ?

Regards,

Sourav

[Tzu-Li Tai]

Hi Sourav,

A little help with more clarification on your last comment.

In sense of "where" the driver program is executed, then yes the Flink driver program runs in a mode similar to Spark's YARN-client.

However, the "role" of the driver program and the work that it is responsible of is quite different between Flink and Spark. In Spark, the driver program is in charge of coordinating Spark workers (executors) and must listen for and accept incoming connections from the workers throughout the job's lifetime. Therefore, in Spark's YARN-client mode, you must keep the driver program process alive otherwise the job will be shutdown.

However, in Flink, the coordination of Flink TaskManagers to complete a job is handled by Flink's JobManager once the client at the driver program submits the job to the JobManager. The driver program is solely used for the job submission and can disconnect afterwards.

Like what Stephan explained, if the user-defined dataflow defines any intermediate results to be retrieved via collect() or print(), the results are transmitted through the JobManager. Only then does the driver program need to stay connected. Note that this connection still does not need to have any connections with the workers (Flink TaskManagers), only the JobManager.

Cheers,
Gordon

[Sourav Mazumder]

Hi Gordon,

Thanks for the explanation. It is much clear now. Looks like a much cleaner approach. In that way the driver program can run in a machine which does not need connectivity to all worker nodes.

Regards,

Sourav

On Mon, Jan 11, 2016 at 9:22 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Sourav,

A little help with more clarification on your last comment.

In sense of "where" the driver program is executed, then yes the Flink
driver program runs in a mode similar to Spark's YARN-client.

However, the "role" of the driver program and the work that it is
responsible of is quite different between Flink and Spark. In Spark, the
driver program is in charge of coordinating Spark workers (executors) and
must listen for and accept incoming connections from the workers throughout
the job's lifetime. Therefore, in Spark's YARN-client mode, you must keep
the driver program process alive otherwise the job will be shutdown.

However, in Flink, the coordination of Flink TaskManagers to complete a job
is handled by Flink's JobManager once the client at the driver program
submits the job to the JobManager. The driver program is solely used for the
job submission and can disconnect afterwards.

Like what Stephan explained, if the user-defined dataflow defines any
intermediate results to be retrieved via collect() or print(), the results
are transmitted through the JobManager. Only then does the driver program
need to stay connected. Note that this connection still does not need to
have any connections with the workers (Flink TaskManagers), only the
JobManager.

Cheers,
Gordon



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-with-Yarn-tp4224p4227.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.