Monitoring Flink on Yarn

[Lydia Ickler]

Hi all, 

I am using Flink 1.1.3 on Yarn and I wanted to ask how I can save the monitoring logs, e.g. for I/O or network, to HDFS or local FS?

Since Yarn closes the Flink session after finishing the job I can't access the log via REST API.

I am looking forward to your answer!

Best regards,

Lydia 

[Shannon Carey]

Check out Logstash or Splunk. Those can pipe your logs into an external database which can be used by a purpose-built UI for examining logs, and as a result it doesn't matter if the original files or machines are still around or not.

From: Lydia Ickler <[hidden email]>
Date: Monday, December 19, 2016 at 7:38 AM
To: <[hidden email]>
Subject: Monitoring Flink on Yarn

Hi all, 

I am using Flink 1.1.3 on Yarn and I wanted to ask how I can save the monitoring logs, e.g. for I/O or network, to HDFS or local FS?

Since Yarn closes the Flink session after finishing the job I can't access the log via REST API.

I am looking forward to your answer!

Best regards,

Lydia