Application logs missing from jobmanager log

[Juho Autio]

The logs logged by my job jar before env.execute can't be found in jobmanager log. I can't find them anywhere else either.

I can see all the usual logs by Flink components in the jobmanager log, though. And in taskmanager log I can see both Flink's internal & my application's logs from the execution time operations. But indeed missing the logs from before env.execute.

I'm out of ideas how to debug this further. Any suggestions, or is this the known behaviour?

I'm submitting my flink jobs in yarn cluster mode. Using Flink 1.5-SNAPSHOT.

[Juho Autio]

Ah, found the place! In my case they seem to be going to /home/hadoop/flink-1.5-SNAPSHOT/log/flink-hadoop-client-ip-10-0-10-29.log (for example).

Any reason why these can't be shown in Flink UI, maybe in jobmanager log?

On Fri, Apr 27, 2018 at 12:13 PM, Juho Autio <[hidden email]> wrote:

The logs logged by my job jar before env.execute can't be found in jobmanager log. I can't find them anywhere else either.

I can see all the usual logs by Flink components in the jobmanager log, though. And in taskmanager log I can see both Flink's internal & my application's logs from the execution time operations. But indeed missing the logs from before env.execute.

I'm out of ideas how to debug this further. Any suggestions, or is this the known behaviour?

I'm submitting my flink jobs in yarn cluster mode. Using Flink 1.5-SNAPSHOT.

[Fabian Hueske-2]

Hi Juho,

I assume that these logs are generated from a different process, i.e., the client process and not the JM or TM process.

Hence, they end up in a different log file and are not covered by the log collection of the UI.

The reason is that this process might also be run on a machine outside of the cluster. So the client log file might not be accessible from the UI process.

It seems like good feature request to include the client logs though.

Would you mind opening a JIRA issue for this?

Thanks, Fabian

2018-04-27 11:27 GMT+02:00 Juho Autio <[hidden email]>:

Ah, found the place! In my case they seem to be going to /home/hadoop/flink-1.5-SNAPSHOT/log/flink-hadoop-client-ip-10-0-10-29.log (for example).

Any reason why these can't be shown in Flink UI, maybe in jobmanager log?

On Fri, Apr 27, 2018 at 12:13 PM, Juho Autio <[hidden email]> wrote:

The logs logged by my job jar before env.execute can't be found in jobmanager log. I can't find them anywhere else either.

I can see all the usual logs by Flink components in the jobmanager log, though. And in taskmanager log I can see both Flink's internal & my application's logs from the execution time operations. But indeed missing the logs from before env.execute.

I'm out of ideas how to debug this further. Any suggestions, or is this the known behaviour?

I'm submitting my flink jobs in yarn cluster mode. Using Flink 1.5-SNAPSHOT.

[Juho Autio]

Thanks Fabian, here's the ticket:

https://issues.apache.org/jira/browse/FLINK-9335

On Wed, May 2, 2018 at 12:53 PM, Fabian Hueske <[hidden email]> wrote:

Hi Juho,

I assume that these logs are generated from a different process, i.e., the client process and not the JM or TM process.

Hence, they end up in a different log file and are not covered by the log collection of the UI.

The reason is that this process might also be run on a machine outside of the cluster. So the client log file might not be accessible from the UI process.

It seems like good feature request to include the client logs though.

Would you mind opening a JIRA issue for this?

Thanks, Fabian

2018-04-27 11:27 GMT+02:00 Juho Autio <[hidden email]>:

Ah, found the place! In my case they seem to be going to /home/hadoop/flink-1.5-SNAPSHOT/log/flink-hadoop-client-ip-10-0-10-29.log (for example).

Any reason why these can't be shown in Flink UI, maybe in jobmanager log?

On Fri, Apr 27, 2018 at 12:13 PM, Juho Autio <[hidden email]> wrote:

The logs logged by my job jar before env.execute can't be found in jobmanager log. I can't find them anywhere else either.

I can see all the usual logs by Flink components in the jobmanager log, though. And in taskmanager log I can see both Flink's internal & my application's logs from the execution time operations. But indeed missing the logs from before env.execute.

I'm out of ideas how to debug this further. Any suggestions, or is this the known behaviour?

I'm submitting my flink jobs in yarn cluster mode. Using Flink 1.5-SNAPSHOT.

[Fabian Hueske-2]

Great, thank you!

2018-05-11 10:31 GMT+02:00 Juho Autio <[hidden email]>:

Thanks Fabian, here's the ticket:

https://issues.apache.org/jira/browse/FLINK-9335

On Wed, May 2, 2018 at 12:53 PM, Fabian Hueske <[hidden email]> wrote:

Hi Juho,

I assume that these logs are generated from a different process, i.e., the client process and not the JM or TM process.

Hence, they end up in a different log file and are not covered by the log collection of the UI.

The reason is that this process might also be run on a machine outside of the cluster. So the client log file might not be accessible from the UI process.

It seems like good feature request to include the client logs though.

Would you mind opening a JIRA issue for this?

Thanks, Fabian

2018-04-27 11:27 GMT+02:00 Juho Autio <[hidden email]>:

Ah, found the place! In my case they seem to be going to /home/hadoop/flink-1.5-SNAPSHOT/log/flink-hadoop-client-ip-10-0-10-29.log (for example).

Any reason why these can't be shown in Flink UI, maybe in jobmanager log?

On Fri, Apr 27, 2018 at 12:13 PM, Juho Autio <[hidden email]> wrote:

The logs logged by my job jar before env.execute can't be found in jobmanager log. I can't find them anywhere else either.

I can see all the usual logs by Flink components in the jobmanager log, though. And in taskmanager log I can see both Flink's internal & my application's logs from the execution time operations. But indeed missing the logs from before env.execute.

I'm out of ideas how to debug this further. Any suggestions, or is this the known behaviour?

I'm submitting my flink jobs in yarn cluster mode. Using Flink 1.5-SNAPSHOT.