Flink Standalone cluster - logging problem

[simpleusr]

We are using standalone cluster and submittig jobs through command line
client.

As stated in
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/logging.html
, we are editing log4j-cli.properties but this does not make any effect?  

Anybody seen that before?

Regards




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Chesnay Schepler]

What exactly are you expecting to happen?

On 08.02.2019 15:06, simpleusr wrote:

> We are using standalone cluster and submittig jobs through command line
> client.
>
> As stated in
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/logging.html
> , we are editing log4j-cli.properties but this does not make any effect?
>
> Anybody seen that before?
>
> Regards
>
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

[simpleusr]

Hi Chesnay,

below is the content for my log4j-cli.properties file. I expect my job logs
(packaged under com.mycompany.xyz to be written to file2 appender. However
no file generated with prefix XYZ. I restarted the cluster , canceled
resubmitted several times but none of them helped.


/
log4j.rootLogger=INFO, file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n

log4j.appender.file2=org.apache.log4j.FileAppender
log4j.appender.file2.file=XYZ-${log.file}
log4j.appender.file2.append=false
log4j.appender.file2.layout=org.apache.log4j.PatternLayout
log4j.appender.file2.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n


# Log output from org.apache.flink.yarn to the console. This is used by the
# CliFrontend class when using a per-job YARN cluster.
log4j.logger.org.apache.flink.yarn=INFO, console
log4j.logger.org.apache.flink.yarn.cli.FlinkYarnSessionCli=INFO, console
log4j.logger.org.apache.hadoop=INFO, console

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n

# suppress the warning that hadoop native libraries are not loaded
(irrelevant for the client)
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=OFF

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,
file


log4j.logger.com.hazelcast=INFO, file2
log4j.logger.com.mycompany.xyz=DEBUG, file2/



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Gary Yao-3]

Hi,

Can you define what you mean by "job logs"? For code that is run on the
cluster, i.e., JM or TM, you should add your config to log4j.properties. The
log4j-cli.properties file is only used by the Flink CLI process.

Best,
Gary

On Mon, Feb 11, 2019 at 7:39 AM simpleusr <[hidden email]> wrote:

Hi Chesnay,

below is the content for my log4j-cli.properties file. I expect my job logs
(packaged under com.mycompany.xyz to be written to file2 appender. However
no file generated with prefix XYZ. I restarted the cluster , canceled
resubmitted several times but none of them helped.


/
log4j.rootLogger=INFO, file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n

log4j.appender.file2=org.apache.log4j.FileAppender
log4j.appender.file2.file=XYZ-${log.file}
log4j.appender.file2.append=false
log4j.appender.file2.layout=org.apache.log4j.PatternLayout
log4j.appender.file2.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n


# Log output from org.apache.flink.yarn to the console. This is used by the
# CliFrontend class when using a per-job YARN cluster.
log4j.logger.org.apache.flink.yarn=INFO, console
log4j.logger.org.apache.flink.yarn.cli.FlinkYarnSessionCli=INFO, console
log4j.logger.org.apache.hadoop=INFO, console

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n

# suppress the warning that hadoop native libraries are not loaded
(irrelevant for the client)
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=OFF

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,
file


log4j.logger.com.hazelcast=INFO, file2
log4j.logger.com.mycompany.xyz=DEBUG, file2/



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[simpleusr]

Hi Gary,

By "job logs" I mean all the loggers under a subpackage of com.mycompany.xyz
.

We are using ./bin/flink run command for job execution thats why I modified
log4j-cli.properties. Modification of log4j.properties also did not help...

Regards



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Could you pls try modifying conf/logback.xml .

Regards,

Selvaraj C

On Mon, Feb 11, 2019 at 4:32 PM simpleusr <[hidden email]> wrote:

Hi Gary,

By "job logs" I mean all the loggers under a subpackage of com.mycompany.xyz
.

We are using ./bin/flink run command for job execution thats why I modified
log4j-cli.properties. Modification of log4j.properties also did not help...

Regards



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

--

[simpleusr]

Hi Selveraj,

This did not help either.

Thanks



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Gary Yao-4]

Hi,

Are you logging from your own operator implementations, and you expect these
log messages to end up in a file prefixed with XYZ-? If that is the case,
modifying log4j-cli.properties will not be expedient as I wrote earlier.

You should modify the log4j.properties on all hosts that are running the
JobManager (JM) and TaskManagers (TM). Consequently, the log files can only be
found on the hosts that are running the JM and TMs.

Furthermore, I see a problem with the following line in your log4j
configuration:

    log4j.appender.file2.file=XYZ-${log.file}

Here, ${log.file} can be an absolute path, which means you would end up
prefixing the whole path instead of just the filename.

If this does not help, please share a minimum working example with us.

Best,
Gary

On Mon, Feb 11, 2019 at 12:02 PM simpleusr <[hidden email]> wrote:

Hi Gary,

By "job logs" I mean all the loggers under a subpackage of com.mycompany.xyz
.

We are using ./bin/flink run command for job execution thats why I modified
log4j-cli.properties. Modification of log4j.properties also did not help...

Regards



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/