Flink WebUI on YARN behind firewall

"Fortunate is he, who is able to know the causes of things." -Virgil

vijikarthi

Re: Flink WebUI on YARN behind firewall

Hi Trevor,

I am seeing similar issue for a JIRA that I am working now. I am yet to trace the Yarn Web UI code to find out how the "tracking URL" is being handled. To ublock, you could use the tracking URL (Flink UI URL) directly to access Flink Web UI to by-pass Yarn UI redirection. You can find the tracking URI in the Job Manager log file from Yarn container.

Regards

Vijay

On Friday, August 26, 2016 10:52 AM, Trevor Grant <[hidden email]> wrote:

I decided it made the most sense to open up a new thread.

I am running Flink on a cluster behind a firewall. Things seem to be working fine, but when I access the YARN web-ui and click on the flink application-ui, i get the jobmanager ui, but it is broken.

It is a broken link to a flink image and

Apache Flink Dashboard

but that seems to be more related to running jobs than accessing the UI

any help would be appreciated

Trevor Grant

Data Scientist

https://github.com/rawkintrevo/bluemix-extra-services

"Fortunate is he, who is able to know the causes of things." -Virgil

Trevor Grant

Re: Flink WebUI on YARN behind firewall

Ahh, sorry I misunderstood.

Your comment provided insight for me though.

To anyone else who is having issues, maybe the following will help them. I was trying to deploy Flink on an IBM BigInsights Cloud cluster (disclaimer, I work for IBM, not trying to promote a company, but they do give me free cluster time so I tend to play there).

At anyrate- flink_example.py is a script that will do it nice and clean for anyone on BigInsights Cloud, and may be useful for other similar cloud YARN deployments so I'll just leave it here...

The solution in short-

1) install flink-

2) run yarn-session.sh

3) parse the stdout from above command (see python script, easy to follow or email me if not)

4) open the ApplicationUI via the Yarn UI, go to the #/submit tab

5) in the submit tab, IF you open with a web browser (bc it relies on javascript) it may give you a hostname:port <- save that port

6) combine the port you just got , with an IP address you got when parsing in step 3

7) open ssh tunneling to that IP:port

Ugly, but it does work.

best

Trevor Grant

Data Scientist

"Fortunate is he, who is able to know the causes of things." -Virgil

On Fri, Aug 26, 2016 at 5:54 PM, Vijay Srinivasaraghavan <[hidden email]> wrote:

Hi Trevor,

Sorry, I think I did not explain it clearly. I am not working on the issue that you have mentioned but I have noticed similar problem for another JIRA that I am currenty working (Flink Web UI basic authentication). The redirection from Yarn UI to Flink does not propogate the authentication headers. I was planning to investigate from Yarn UI as to what exactly is happening. Please go-ahead and create a JIRA anyway and someone will pick it up.

Regards
Vijay

On Friday, August 26, 2016 1:34 PM, Trevor Grant <[hidden email]> wrote:

That's awesome Vijay, thanks for the work- will definitely help (esp as Flink gets picked up for more cloud cluster deployments)!

Do you have the JIRA so I can watch it?

Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/users/3002022/rawkintrevo
http://trevorgrant.org

"Fortunate is he, who is able to know the causes of things." -Virgil

On Fri, Aug 26, 2016 at 1:14 PM, Vijay Srinivasaraghavan <[hidden email]> wrote:
Hi Trevor,

I am seeing similar issue for a JIRA that I am working now. I am yet to trace the Yarn Web UI code to find out how the "tracking URL" is being handled. To ublock, you could use the tracking URL (Flink UI URL) directly to access Flink Web UI to by-pass Yarn UI redirection. You can find the tracking URI in the Job Manager log file from Yarn container.

Regards
Vijay

On Friday, August 26, 2016 10:52 AM, Trevor Grant <[hidden email]> wrote:

I decided it made the most sense to open up a new thread.

I am running Flink on a cluster behind a firewall. Things seem to be working fine, but when I access the YARN web-ui and click on the flink application-ui, i get the jobmanager ui, but it is broken.

It is a broken link to a flink image and
Apache Flink Dashboard
Overview
Running Jobs
Completed Jobs
Task Managers
Job Manager
Submit new Job
but none of the links work, no images no pretty formatting. does anyone have a quick idea why this would be? I found http://mail-archives. apache.org/mod_mbox/flink- user/201511.mbox/%3CCAGr9p8B_ aKhbjLqVQQUZeO_eSz6P=eWKP+ Kg1sQ65SB0npsX2Q@.... com%3E

but that seems to be more related to running jobs than accessing the UI

any help would be appreciated

tg

Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/ users/3002022/rawkintrevo
http://trevorgrant.org

"Fortunate is he, who is able to know the causes of things." -Virgil

Shannon Carey

Re: Flink WebUI on YARN behind firewall

In reply to this post by Trevor Grant

I've noticed this too. The UI incorrectly tries to load its resources (CSS, JS) from root. Try adding a trailing slash to the URL.

-Shannon

From: Trevor Grant <[hidden email]>
Date: Friday, August 26, 2016 at 12:52 PM
To: <[hidden email]>
Subject: Flink WebUI on YARN behind firewall

I decided it made the most sense to open up a new thread.

I am running Flink on a cluster behind a firewall. Things seem to be working fine, but when I access the YARN web-ui and click on the flink application-ui, i get the jobmanager ui, but it is broken.

It is a broken link to a flink image and

Apache Flink Dashboard

Overview
Running Jobs
Completed Jobs
Task Managers
Job Manager
Submit new Job

but that seems to be more related to running jobs than accessing the UI

any help would be appreciated

Trevor Grant

Data Scientist