Flink on yarn : yarn-session understanding

[anuj.aj07]

I am running some stream jobs that are long-running always. I am currently submitting each job as a standalone job on yarn. 

1. I need to understand what is the advantage of using yarn-session and when should I use that. 

2. Also, I am not able to access rest API services is it because I am running as standalone job over yarn. Is REST API works only in yarn-session?

--

[Xintong Song]

Hi Anuj,

By "standalone job on yarn", I assume you mean running one job per Flink cluster on Yarn, which is also known as job mode, or per-job mode? I'm asking because Flink has another standalone deployment mode [1], aside from the Yarn deployment mode.

1. The major difference between Flink Application Cluster (a.k.a. job mode) [2]  and Flink Session Cluster [3] is whether you can execute only multiple jobs per Flink cluster. A session cluster allows you to execute multiple jobs in the same Flink cluster, sharing/reusing the process resources. You might want to run a session cluster for various purposes. One example is to reduce the resource overhead of Flink Master processes. When you have multiple jobs with very small scale, having one Master process per job could become a significant overhead. Another use case is when you need to rapidly submit new jobs (e.g., interactive analysis), you may want to leverage the pre-exist task manager processes to speed up the initial process.

2. AFAIK, the REST API should work for both application & session clusters, except for that you cannot submit new jobs to an application cluster since it's designed for executing only one job.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/cluster_setup.html

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.10/concepts/glossary.html#flink-application-cluster

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.10/concepts/glossary.html#flink-session-cluster

On Mon, Jun 8, 2020 at 9:39 PM aj <[hidden email]> wrote:

I am running some stream jobs that are long-running always. I am currently submitting each job as a standalone job on yarn. 

1. I need to understand what is the advantage of using yarn-session and when should I use that. 

2. Also, I am not able to access rest API services is it because I am running as standalone job over yarn. Is REST API works only in yarn-session?

--

Thanks & Regards,
Anuj Jain

[Andrey Zagrebin-5]

Hi Anuj,

Afaik, the REST API should work for both modes. What is the issue? Maybe, some network problem to connect to YARN application master?

Best,
Andrey

On Mon, Jun 8, 2020 at 4:39 PM aj <[hidden email]> wrote:

I am running some stream jobs that are long-running always. I am currently submitting each job as a standalone job on yarn. 

1. I need to understand what is the advantage of using yarn-session and when should I use that. 

2. Also, I am not able to access rest API services is it because I am running as standalone job over yarn. Is REST API works only in yarn-session?

--

Thanks & Regards,
Anuj Jain

[anuj.aj07]

Ok, thanks for the clarification on yarn session. 

 I am trying to connect to job manager on 8081 but it's not connecting. 

So this is the address shown on my Flink job UI and i am trying to connect rest address on 8081 but its refusing connection.

On Tue, Jun 9, 2020 at 1:03 PM Andrey Zagrebin <[hidden email]> wrote:

Hi Anuj,

Afaik, the REST API should work for both modes. What is the issue? Maybe, some network problem to connect to YARN application master?

Best,
Andrey

On Mon, Jun 8, 2020 at 4:39 PM aj <[hidden email]> wrote:

I am running some stream jobs that are long-running always. I am currently submitting each job as a standalone job on yarn. 

1. I need to understand what is the advantage of using yarn-session and when should I use that. 

2. Also, I am not able to access rest API services is it because I am running as standalone job over yarn. Is REST API works only in yarn-session?

--

Thanks & Regards,
Anuj Jain

--

[Vikash Dat]

yarn will assign a random port when flink is deployed. To get the port you need to do a yarn application -list and see the tracking url assigned to your flink cluster. The port in that url will be the port you need to use for the rest api. 

On Tue, Jun 16, 2020 at 08:49 aj <[hidden email]> wrote:

Ok, thanks for the clarification on yarn session. 

 I am trying to connect to job manager on 8081 but it's not connecting. 

So this is the address shown on my Flink job UI and i am trying to connect rest address on 8081 but its refusing connection.

On Tue, Jun 9, 2020 at 1:03 PM Andrey Zagrebin <[hidden email]> wrote:

Hi Anuj,

Afaik, the REST API should work for both modes. What is the issue? Maybe, some network problem to connect to YARN application master?

Best,
Andrey

On Mon, Jun 8, 2020 at 4:39 PM aj <[hidden email]> wrote:

I am running some stream jobs that are long-running always. I am currently submitting each job as a standalone job on yarn. 

1. I need to understand what is the advantage of using yarn-session and when should I use that. 

2. Also, I am not able to access rest API services is it because I am running as standalone job over yarn. Is REST API works only in yarn-session?

--

Thanks & Regards,
Anuj Jain

--

Thanks & Regards,
Anuj Jain

Mob. : [phone]
Skype : anuj.jain07