Flink Session TM Logs

[Richard Moorhead]

When running a flink session on YARN, task manager logs for a job are not available after completion. How do we locate these logs?

[Richard Moorhead]

---------- Forwarded message ---------
From: Robert Metzger <[hidden email]>
Date: Fri, Jul 24, 2020 at 1:09 PM
Subject: Re: Flink Session TM Logs
To: Richard Moorhead <[hidden email]>

I accidentally replied to you directly, not to the user@ mailing list. Is it okay for you to publish the thread on the list again?

On Fri, Jul 24, 2020 at 8:01 PM Richard Moorhead <[hidden email]> wrote:

It is enabled. The issue is that for a long running flink session -which may execute many jobs- the task managers, after a job is completed, are gone, and their logs arent available.

What I have noticed is that when the session is terminated I am able to find the logs in the job history server under the associated yarn application id.

On Fri, Jul 24, 2020 at 12:51 PM Robert Metzger <[hidden email]> wrote:

Hi Richard,

you need to enable YARN log aggregation to access logs of finished YARN applications.

On Fri, Jul 24, 2020 at 5:58 PM Richard Moorhead <[hidden email]> wrote:

When running a flink session on YARN, task manager logs for a job are not available after completion. How do we locate these logs?

[Yangze Guo]

Hi, Richard

Before the session has been terminated, you could not fetch the
terminated TM logs. One possible solution could be leveraging the
log4j2 appenders[1]. Flink uses log4j2 as default in the latest
release 1.11.

[1] https://logging.apache.org/log4j/2.x/manual/appenders.html

Best,
Yangze Guo

On Sat, Jul 25, 2020 at 2:37 AM Richard Moorhead
<[hidden email]> wrote:

>
>
>
> ---------- Forwarded message ---------
> From: Robert Metzger <[hidden email]>
> Date: Fri, Jul 24, 2020 at 1:09 PM
> Subject: Re: Flink Session TM Logs
> To: Richard Moorhead <[hidden email]>
>
>
> I accidentally replied to you directly, not to the user@ mailing list. Is it okay for you to publish the thread on the list again?
>
>
>
> On Fri, Jul 24, 2020 at 8:01 PM Richard Moorhead <[hidden email]> wrote:
>>
>> It is enabled. The issue is that for a long running flink session -which may execute many jobs- the task managers, after a job is completed, are gone, and their logs arent available.
>>
>> What I have noticed is that when the session is terminated I am able to find the logs in the job history server under the associated yarn application id.
>>
>> On Fri, Jul 24, 2020 at 12:51 PM Robert Metzger <[hidden email]> wrote:
>>>
>>> Hi Richard,
>>>
>>> you need to enable YARN log aggregation to access logs of finished YARN applications.
>>>
>>> On Fri, Jul 24, 2020 at 5:58 PM Richard Moorhead <[hidden email]> wrote:
>>>>
>>>> When running a flink session on YARN, task manager logs for a job are not available after completion. How do we locate these logs?
>>>>

[rmetzger0]

Hi Richard,

thanks for forwarding my answer to the list!

I fear that Flink does not have a built-in solution for serving the logs of a finished TaskManager while a YARN session is still running.

I agree with Yangze that you probably have to rely on an external logging service, such as ElasticSearch or Splunk to index your log events.

Maybe there's also some tooling from Cloudera specifically made for YARN?

On Mon, Jul 27, 2020 at 3:46 AM Yangze Guo <[hidden email]> wrote:

Hi, Richard

Before the session has been terminated, you could not fetch the
terminated TM logs. One possible solution could be leveraging the
log4j2 appenders[1]. Flink uses log4j2 as default in the latest
release 1.11.

[1] https://logging.apache.org/log4j/2.x/manual/appenders.html

Best,
Yangze Guo

On Sat, Jul 25, 2020 at 2:37 AM Richard Moorhead
<[hidden email]> wrote:
>
>
>
> ---------- Forwarded message ---------
> From: Robert Metzger <[hidden email]>
> Date: Fri, Jul 24, 2020 at 1:09 PM
> Subject: Re: Flink Session TM Logs
> To: Richard Moorhead <[hidden email]>
>
>
> I accidentally replied to you directly, not to the user@ mailing list. Is it okay for you to publish the thread on the list again?
>
>
>
> On Fri, Jul 24, 2020 at 8:01 PM Richard Moorhead <[hidden email]> wrote:
>>
>> It is enabled. The issue is that for a long running flink session -which may execute many jobs- the task managers, after a job is completed, are gone, and their logs arent available.
>>
>> What I have noticed is that when the session is terminated I am able to find the logs in the job history server under the associated yarn application id.
>>
>> On Fri, Jul 24, 2020 at 12:51 PM Robert Metzger <[hidden email]> wrote:
>>>
>>> Hi Richard,
>>>
>>> you need to enable YARN log aggregation to access logs of finished YARN applications.
>>>
>>> On Fri, Jul 24, 2020 at 5:58 PM Richard Moorhead <[hidden email]> wrote:
>>>>
>>>> When running a flink session on YARN, task manager logs for a job are not available after completion. How do we locate these logs?
>>>>

[Yang Wang]

Just share another method about how to access the finished TaskManager logs on Yarn.

Currently, only when a Yarn application finished/failed/killed, the logs will be aggregated to HDFS. That means

if the Flink application is still running, you could still use the Yarn NodeManager webUI to access the TM logs. Please

build the url in the following schema.

http://<NodeManagerAddress:Port>/node/containerlogs/container_xxxxxx/<User>/taskmanager.log

Best,

Yang

Robert Metzger <[hidden email]> 于2020年7月27日周一 下午2:42写道：

Hi Richard,

thanks for forwarding my answer to the list!

I fear that Flink does not have a built-in solution for serving the logs of a finished TaskManager while a YARN session is still running.

I agree with Yangze that you probably have to rely on an external logging service, such as ElasticSearch or Splunk to index your log events.

Maybe there's also some tooling from Cloudera specifically made for YARN?

On Mon, Jul 27, 2020 at 3:46 AM Yangze Guo <[hidden email]> wrote:

Hi, Richard

Before the session has been terminated, you could not fetch the
terminated TM logs. One possible solution could be leveraging the
log4j2 appenders[1]. Flink uses log4j2 as default in the latest
release 1.11.

[1] https://logging.apache.org/log4j/2.x/manual/appenders.html

Best,
Yangze Guo

On Sat, Jul 25, 2020 at 2:37 AM Richard Moorhead
<[hidden email]> wrote:
>
>
>
> ---------- Forwarded message ---------
> From: Robert Metzger <[hidden email]>
> Date: Fri, Jul 24, 2020 at 1:09 PM
> Subject: Re: Flink Session TM Logs
> To: Richard Moorhead <[hidden email]>
>
>
> I accidentally replied to you directly, not to the user@ mailing list. Is it okay for you to publish the thread on the list again?
>
>
>
> On Fri, Jul 24, 2020 at 8:01 PM Richard Moorhead <[hidden email]> wrote:
>>
>> It is enabled. The issue is that for a long running flink session -which may execute many jobs- the task managers, after a job is completed, are gone, and their logs arent available.
>>
>> What I have noticed is that when the session is terminated I am able to find the logs in the job history server under the associated yarn application id.
>>
>> On Fri, Jul 24, 2020 at 12:51 PM Robert Metzger <[hidden email]> wrote:
>>>
>>> Hi Richard,
>>>
>>> you need to enable YARN log aggregation to access logs of finished YARN applications.
>>>
>>> On Fri, Jul 24, 2020 at 5:58 PM Richard Moorhead <[hidden email]> wrote:
>>>>
>>>> When running a flink session on YARN, task manager logs for a job are not available after completion. How do we locate these logs?
>>>>

[范超]

Hi Rechard

Maybe you can try using cli

“yarn logs –applicationId yourYarnAppId”

to check your logs or just to find your app logs in the yarn webui

 

发件人: Richard Moorhead [mailto:[hidden email]]
发送时间: 2020年7月24日 星期五 23:55
收件人: user <[hidden email]>
主题: Flink Session TM Logs

 

When running a flink session on YARN, task manager logs for a job are not available after completion. How do we locate these logs?