Can't list logs or stdout through web console on Flink 1.7 Kubernetes

[William Saar-2]

I'm running Flink 1.7 in ECS, is this a known issue or should I create a jira?

The web console doesn't show anything when trying to list logs or stdout for task managers and the job manager log have stack traces for the errors

2018-12-19 15:35:53,498 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler  - Failed to transfer file from TaskExecutor d7fd266047d5acfaddeb1156bdb23ff3.
java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file STDOUT is not available on the TaskExecutor.

2018-12-19 15:36:02,538 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler  - Failed to transfer file from TaskExecutor d7fd266047d5acfaddeb1156bdb23ff3.
java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG is not available on the TaskExecutor.

[Steven Nelson]

There is a known issue for this I believe. The problem is that the containerized versions of Flink output logs to STDOUT instead of files inside the node. If you pull use docker logs on the container you can see what you’re looking for. I use the Kube dashboard to view the logs centrally.

Sent from my iPhone

> On Dec 19, 2018, at 9:40 AM, William Saar <[hidden email]> wrote:
>
>
> I'm running Flink 1.7 in ECS, is this a known issue or should I create a jira?
>
> The web console doesn't show anything when trying to list logs or stdout for task managers and the job manager log have stack traces for the errors
> 2018-12-19 15:35:53,498 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler  - Failed to transfer file from TaskExecutor d7fd266047d5acfaddeb1156bdb23ff3.
> java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file STDOUT is not available on the TaskExecutor.
>
> 2018-12-19 15:36:02,538 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler  - Failed to transfer file from TaskExecutor d7fd266047d5acfaddeb1156bdb23ff3.
> java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG is not available on the TaskExecutor.
>

[Joshua Fan]

wow, I met similar situation using flink 1.7 on yarn.

there was no jobmanager.log on the node but jobmanager.out and jobmanager.error, and jobmanager.error contains the log message. so , there was nothing in the webUI.

I do not know why this happened. by the way, I used logback to do log staff.

On Thu, Dec 20, 2018 at 12:50 AM Steven Nelson <[hidden email]> wrote:

There is a known issue for this I believe. The problem is that the containerized versions of Flink output logs to STDOUT instead of files inside the node. If you pull use docker logs on the container you can see what you’re looking for. I use the Kube dashboard to view the logs centrally.

Sent from my iPhone

> On Dec 19, 2018, at 9:40 AM, William Saar <[hidden email]> wrote:
>
>
> I'm running Flink 1.7 in ECS, is this a known issue or should I create a jira?
>
> The web console doesn't show anything when trying to list logs or stdout for task managers and the job manager log have stack traces for the errors
> 2018-12-19 15:35:53,498 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler  - Failed to transfer file from TaskExecutor d7fd266047d5acfaddeb1156bdb23ff3.
> java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file STDOUT is not available on the TaskExecutor.
>
> 2018-12-19 15:36:02,538 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler  - Failed to transfer file from TaskExecutor d7fd266047d5acfaddeb1156bdb23ff3.
> java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG is not available on the TaskExecutor.
>

[Chesnay Schepler]

@Steven: Do you happen do know whether a JIRA exists for this issue?

@Joshua: Does this also happen if you use log4j?

On 26.12.2018 11:33, Joshua Fan wrote:

wow, I met similar situation using flink 1.7 on yarn.

there was no jobmanager.log on the node but jobmanager.out and jobmanager.error, and jobmanager.error contains the log message. so , there was nothing in the webUI.

I do not know why this happened. by the way, I used logback to do log staff.

On Thu, Dec 20, 2018 at 12:50 AM Steven Nelson <[hidden email]> wrote:

There is a known issue for this I believe. The problem is that the containerized versions of Flink output logs to STDOUT instead of files inside the node. If you pull use docker logs on the container you can see what you’re looking for. I use the Kube dashboard to view the logs centrally.

Sent from my iPhone

> On Dec 19, 2018, at 9:40 AM, William Saar <[hidden email]> wrote:
>
>
> I'm running Flink 1.7 in ECS, is this a known issue or should I create a jira?
>
> The web console doesn't show anything when trying to list logs or stdout for task managers and the job manager log have stack traces for the errors
> 2018-12-19 15:35:53,498 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler  - Failed to transfer file from TaskExecutor d7fd266047d5acfaddeb1156bdb23ff3.
> java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file STDOUT is not available on the TaskExecutor.
>
> 2018-12-19 15:36:02,538 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler  - Failed to transfer file from TaskExecutor d7fd266047d5acfaddeb1156bdb23ff3.
> java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG is not available on the TaskExecutor.
>

[Joshua Fan]

I found the root cause why log-not-show when use logback, because flink does not include the logback-*.jar in the lib folder.

After I put the logback jar file in lib, everything is ok now.

On Fri, Dec 28, 2018 at 10:41 PM Chesnay Schepler <[hidden email]> wrote:

@Steven: Do you happen do know whether a JIRA exists for this issue?

@Joshua: Does this also happen if you use log4j?

On 26.12.2018 11:33, Joshua Fan wrote:

wow, I met similar situation using flink 1.7 on yarn.

there was no jobmanager.log on the node but jobmanager.out and jobmanager.error, and jobmanager.error contains the log message. so , there was nothing in the webUI.

I do not know why this happened. by the way, I used logback to do log staff.

On Thu, Dec 20, 2018 at 12:50 AM Steven Nelson <[hidden email]> wrote:

There is a known issue for this I believe. The problem is that the containerized versions of Flink output logs to STDOUT instead of files inside the node. If you pull use docker logs on the container you can see what you’re looking for. I use the Kube dashboard to view the logs centrally.

Sent from my iPhone

> On Dec 19, 2018, at 9:40 AM, William Saar <[hidden email]> wrote:
>
>
> I'm running Flink 1.7 in ECS, is this a known issue or should I create a jira?
>
> The web console doesn't show anything when trying to list logs or stdout for task managers and the job manager log have stack traces for the errors
> 2018-12-19 15:35:53,498 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler  - Failed to transfer file from TaskExecutor d7fd266047d5acfaddeb1156bdb23ff3.
> java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file STDOUT is not available on the TaskExecutor.
>
> 2018-12-19 15:36:02,538 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler  - Failed to transfer file from TaskExecutor d7fd266047d5acfaddeb1156bdb23ff3.
> java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG is not available on the TaskExecutor.
>