Using memory logging in Flink

[Filip Łęczycki]

Hi,

I am trying to enable logging of memory usage on flink 0.10.0 by adding:

taskmanager.debug.memory.startLogThread: true

to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?

Best regards/Pozdrawiam,

[Stephan Ewen]

Hi!

That is exactly the right way to do it. Logging has to be at least INFO and the parameter "taskmanager.debug.memory.startLogThread" set to true.

The log output should be under "org.apache.flink.runtime.taskmanager.TaskManager".

Do you see other outputs for that class in the log?

Make sure you restarted the TaskManager processes after you changed the config file.

Greetings,

Stephan

On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <[hidden email]> wrote:

Hi,

I am trying to enable logging of memory usage on flink 0.10.0 by adding:

taskmanager.debug.memory.startLogThread: true

to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?

Best regards/Pozdrawiam,

Filip Łęczycki

[Filip Łęczycki]

Hi,

Thank you for your reply!

I have made sure I restarted the TaskManager after changing config, but it didn't resolve the issue.The config is loaded as I can see the following line in the log:

09:12:2015 00:00:21,894 DEBUG org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.debug.memory.startLogThread, true

I am running a job on local standalone flink instance and my log4j.properties look like this:

log4j.rootLogger=DEBUG, file

# Log all infos in the given file

log4j.appender.file=org.apache.log4j.FileAppender

log4j.appender.file.file=${log.file}

log4j.appender.file.append=false

log4j.appender.file.layout=org.apache.log4j.PatternLayout

log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler

log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

Maybe I can set some property in the job's code so that it would force such verbose logging?  I need those logs to run some flink performance tests but maybe I can somehow extract the benchmark results without them,  do you know any other way to monitor Flink Job's memory usage and GC time, other than looking at web interface?

Best regards,

Filip Łęczycki

2015-12-08 20:48 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi!

That is exactly the right way to do it. Logging has to be at least INFO and the parameter "taskmanager.debug.memory.startLogThread" set to true.

The log output should be under "org.apache.flink.runtime.taskmanager.TaskManager".

Do you see other outputs for that class in the log?

Make sure you restarted the TaskManager processes after you changed the config file.

Greetings,

Stephan

On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <[hidden email]> wrote:

Hi,

I am trying to enable logging of memory usage on flink 0.10.0 by adding:

taskmanager.debug.memory.startLogThread: true

to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?

Best regards/Pozdrawiam,

Filip Łęczycki

[Till Rohrmann]

I assume you're looking in the taskmanager log file for the memory usage logging statements, right?

Cheers,

Till

On Wed, Dec 9, 2015 at 12:15 AM, Filip Łęczycki <[hidden email]> wrote:

Hi,

Thank you for your reply!

I have made sure I restarted the TaskManager after changing config, but it didn't resolve the issue.The config is loaded as I can see the following line in the log:

09:12:2015 00:00:21,894 DEBUG org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.debug.memory.startLogThread, true

I am running a job on local standalone flink instance and my log4j.properties look like this:

log4j.rootLogger=DEBUG, file

# Log all infos in the given file

log4j.appender.file=org.apache.log4j.FileAppender

log4j.appender.file.file=${log.file}

log4j.appender.file.append=false

log4j.appender.file.layout=org.apache.log4j.PatternLayout

log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler

log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

Maybe I can set some property in the job's code so that it would force such verbose logging?  I need those logs to run some flink performance tests but maybe I can somehow extract the benchmark results without them,  do you know any other way to monitor Flink Job's memory usage and GC time, other than looking at web interface?

Best regards,

Filip Łęczycki

Pozdrawiam,
Filip Łęczycki

2015-12-08 20:48 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi!

That is exactly the right way to do it. Logging has to be at least INFO and the parameter "taskmanager.debug.memory.startLogThread" set to true.

The log output should be under "org.apache.flink.runtime.taskmanager.TaskManager".

Do you see other outputs for that class in the log?

Make sure you restarted the TaskManager processes after you changed the config file.

Greetings,

Stephan

On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <[hidden email]> wrote:

Hi,

I am trying to enable logging of memory usage on flink 0.10.0 by adding:

taskmanager.debug.memory.startLogThread: true

to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?

Best regards/Pozdrawiam,

Filip Łęczycki

[Stephan Ewen]

Hi Filip!

Someone else just used the memory logging with the exact described settings - it worked.

There is probably some mixup, you may be looking into the wrong log file, or may setting the a value in a different config...

Stephan

On Wed, Dec 9, 2015 at 10:58 AM, Till Rohrmann <[hidden email]> wrote:

I assume you're looking in the taskmanager log file for the memory usage logging statements, right?

Cheers,

Till

On Wed, Dec 9, 2015 at 12:15 AM, Filip Łęczycki <[hidden email]> wrote:

Hi,

Thank you for your reply!

I have made sure I restarted the TaskManager after changing config, but it didn't resolve the issue.The config is loaded as I can see the following line in the log:

09:12:2015 00:00:21,894 DEBUG org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.debug.memory.startLogThread, true

I am running a job on local standalone flink instance and my log4j.properties look like this:

log4j.rootLogger=DEBUG, file

# Log all infos in the given file

log4j.appender.file=org.apache.log4j.FileAppender

log4j.appender.file.file=${log.file}

log4j.appender.file.append=false

log4j.appender.file.layout=org.apache.log4j.PatternLayout

log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler

log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

Maybe I can set some property in the job's code so that it would force such verbose logging?  I need those logs to run some flink performance tests but maybe I can somehow extract the benchmark results without them,  do you know any other way to monitor Flink Job's memory usage and GC time, other than looking at web interface?

Best regards,

Filip Łęczycki

Pozdrawiam,
Filip Łęczycki

2015-12-08 20:48 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi!

That is exactly the right way to do it. Logging has to be at least INFO and the parameter "taskmanager.debug.memory.startLogThread" set to true.

The log output should be under "org.apache.flink.runtime.taskmanager.TaskManager".

Do you see other outputs for that class in the log?

Make sure you restarted the TaskManager processes after you changed the config file.

Greetings,

Stephan

On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <[hidden email]> wrote:

Hi,

I am trying to enable logging of memory usage on flink 0.10.0 by adding:

taskmanager.debug.memory.startLogThread: true

to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?

Best regards/Pozdrawiam,

Filip Łęczycki

[Filip Łęczycki]

Hi Stephan, Till

I am looking for the memory usage info in flink_<user>_jobmanager... log. As there are no other log files in /log dir I assumed this is the right place to look. I will try to setup a new clean instance and check whether it will work. 

Thank you very much for your assistance!

Best regards,

Filip Łęczycki

2015-12-09 11:13 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi Filip!

Someone else just used the memory logging with the exact described settings - it worked.

There is probably some mixup, you may be looking into the wrong log file, or may setting the a value in a different config...

Stephan

On Wed, Dec 9, 2015 at 10:58 AM, Till Rohrmann <[hidden email]> wrote:

I assume you're looking in the taskmanager log file for the memory usage logging statements, right?

Cheers,

Till

On Wed, Dec 9, 2015 at 12:15 AM, Filip Łęczycki <[hidden email]> wrote:

Hi,

Thank you for your reply!

I have made sure I restarted the TaskManager after changing config, but it didn't resolve the issue.The config is loaded as I can see the following line in the log:

09:12:2015 00:00:21,894 DEBUG org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.debug.memory.startLogThread, true

I am running a job on local standalone flink instance and my log4j.properties look like this:

log4j.rootLogger=DEBUG, file

# Log all infos in the given file

log4j.appender.file=org.apache.log4j.FileAppender

log4j.appender.file.file=${log.file}

log4j.appender.file.append=false

log4j.appender.file.layout=org.apache.log4j.PatternLayout

log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler

log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

Maybe I can set some property in the job's code so that it would force such verbose logging?  I need those logs to run some flink performance tests but maybe I can somehow extract the benchmark results without them,  do you know any other way to monitor Flink Job's memory usage and GC time, other than looking at web interface?

Best regards,

Filip Łęczycki

Pozdrawiam,
Filip Łęczycki

2015-12-08 20:48 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi!

That is exactly the right way to do it. Logging has to be at least INFO and the parameter "taskmanager.debug.memory.startLogThread" set to true.

The log output should be under "org.apache.flink.runtime.taskmanager.TaskManager".

Do you see other outputs for that class in the log?

Make sure you restarted the TaskManager processes after you changed the config file.

Greetings,

Stephan

On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <[hidden email]> wrote:

Hi,

I am trying to enable logging of memory usage on flink 0.10.0 by adding:

taskmanager.debug.memory.startLogThread: true

to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?

Best regards/Pozdrawiam,

Filip Łęczycki

[Till Rohrmann]

How do you start the flink cluster? If it's a standalone cluster and you don't use a shared directory, then you'll find the log of the taskmanager on the machine on which the taskmanager runs. If you use YARN then you can activate log aggregation to retrieve the log easily after the job has finished.

Cheers,

Till

On Wed, Dec 9, 2015 at 2:30 PM, Filip Łęczycki <[hidden email]> wrote:

Hi Stephan, Till

I am looking for the memory usage info in flink_<user>_jobmanager... log. As there are no other log files in /log dir I assumed this is the right place to look. I will try to setup a new clean instance and check whether it will work. 

Thank you very much for your assistance!

Best regards,

Filip Łęczycki

Pozdrawiam,
Filip Łęczycki

2015-12-09 11:13 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi Filip!

Someone else just used the memory logging with the exact described settings - it worked.

There is probably some mixup, you may be looking into the wrong log file, or may setting the a value in a different config...

Stephan

On Wed, Dec 9, 2015 at 10:58 AM, Till Rohrmann <[hidden email]> wrote:

I assume you're looking in the taskmanager log file for the memory usage logging statements, right?

Cheers,

Till

On Wed, Dec 9, 2015 at 12:15 AM, Filip Łęczycki <[hidden email]> wrote:

Hi,

Thank you for your reply!

I have made sure I restarted the TaskManager after changing config, but it didn't resolve the issue.The config is loaded as I can see the following line in the log:

09:12:2015 00:00:21,894 DEBUG org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.debug.memory.startLogThread, true

I am running a job on local standalone flink instance and my log4j.properties look like this:

log4j.rootLogger=DEBUG, file

# Log all infos in the given file

log4j.appender.file=org.apache.log4j.FileAppender

log4j.appender.file.file=${log.file}

log4j.appender.file.append=false

log4j.appender.file.layout=org.apache.log4j.PatternLayout

log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler

log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

Maybe I can set some property in the job's code so that it would force such verbose logging?  I need those logs to run some flink performance tests but maybe I can somehow extract the benchmark results without them,  do you know any other way to monitor Flink Job's memory usage and GC time, other than looking at web interface?

Best regards,

Filip Łęczycki

Pozdrawiam,
Filip Łęczycki

2015-12-08 20:48 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi!

That is exactly the right way to do it. Logging has to be at least INFO and the parameter "taskmanager.debug.memory.startLogThread" set to true.

The log output should be under "org.apache.flink.runtime.taskmanager.TaskManager".

Do you see other outputs for that class in the log?

Make sure you restarted the TaskManager processes after you changed the config file.

Greetings,

Stephan

On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <[hidden email]> wrote:

Hi,

I am trying to enable logging of memory usage on flink 0.10.0 by adding:

taskmanager.debug.memory.startLogThread: true

to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?

Best regards/Pozdrawiam,

Filip Łęczycki