Re: Using memory logging in Flink

Posted by Filip Łęczycki on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Using-memory-logging-in-Flink-tp3970p3978.html

Hi,

Thank you for your reply!

I have made sure I restarted the TaskManager after changing config, but it didn't resolve the issue.The config is loaded as I can see the following line in the log:

09:12:2015 00:00:21,894 DEBUG org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.debug.memory.startLogThread, true

I am running a job on local standalone flink instance and my log4j.properties look like this:

log4j.rootLogger=DEBUG, file

# Log all infos in the given file

log4j.appender.file=org.apache.log4j.FileAppender

log4j.appender.file.file=${log.file}

log4j.appender.file.append=false

log4j.appender.file.layout=org.apache.log4j.PatternLayout

log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler

log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

Maybe I can set some property in the job's code so that it would force such verbose logging?  I need those logs to run some flink performance tests but maybe I can somehow extract the benchmark results without them,  do you know any other way to monitor Flink Job's memory usage and GC time, other than looking at web interface?

Best regards,

Filip Łęczycki

2015-12-08 20:48 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi!

That is exactly the right way to do it. Logging has to be at least INFO and the parameter "taskmanager.debug.memory.startLogThread" set to true.

The log output should be under "org.apache.flink.runtime.taskmanager.TaskManager".

Do you see other outputs for that class in the log?

Make sure you restarted the TaskManager processes after you changed the config file.

Greetings,

Stephan

On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <[hidden email]> wrote:

Hi,

I am trying to enable logging of memory usage on flink 0.10.0 by adding:

taskmanager.debug.memory.startLogThread: true

to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?

Best regards/Pozdrawiam,

Filip Łęczycki