Flink cluster log organization

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink cluster log organization

Soheil Pourbafrani
Hi,

I have a Flink multinode cluster and I use Flink standalone scheduler to deploy applications on the cluster. When I deploy applications on the cluster I can see some log files on the path FLINK_HOME/logs will be created but there is no separate log file for each application and all application logs are merged into files. I have no idea how Flink organize logs on standalone mode. I was wondering if it's possible to have a separate log file for each application? or for example set the log file name and path when I submit the applications on the cluster?
Reply | Threaded
Open this post in threaded view
|

Re: Flink cluster log organization

Xintong Song
Hi Soheil,

Do you mean you are running multiple jobs in the same Flink standalone cluster? Or running multiple Flink standalone clusters on the same set of machines?

For the former, I don't think there is an easy way to separate the logs. The log file contains logs of common framework activities that do not belong to any particular job.

For the latter, there are two ways that I can think of to specify log file path for a Flink cluster. 
  • Set evn.log.dir in flink-conf.yaml
  • Modify log4j.appender.file.file in log4j.properties
Hope that helps.

Thank you~

Xintong Song



On Tue, May 21, 2019 at 7:03 AM Soheil Pourbafrani <[hidden email]> wrote:
Hi,

I have a Flink multinode cluster and I use Flink standalone scheduler to deploy applications on the cluster. When I deploy applications on the cluster I can see some log files on the path FLINK_HOME/logs will be created but there is no separate log file for each application and all application logs are merged into files. I have no idea how Flink organize logs on standalone mode. I was wondering if it's possible to have a separate log file for each application? or for example set the log file name and path when I submit the applications on the cluster?