Logging in Flink 0.9.0-milestone-1

[Stefan Bunk]

Hi Flinkers,

I just switched to 0.9.0-milestone-1, and now I get Flink's logging output again in my console (local execution).

I have a log4j.xml under src/main/resources, which says not to log any Flink job progress updates, and which worked fine so far:

[...]

<logger name="org.apache.flink">

                <level value="ERROR" />

                <appender-ref ref="Console" />

</logger>

[...]

See the full file at [1].

However, this file seems to be ignored now. What do I need to change?

Cheers

Stefan

[1] https://gist.github.com/knub/bb1b3ffc6b47e2a1376d

[rmetzger0]

Hi Stefan,

we made a stupid mistake in the 0.9.0-milestone-1 release by including our log4j.properties into the flink-runtime jar. its also in the fat jar in flink-dist.

Maybe you can pass the name of your log4j file to your application with -Dlog4j.configuration=log4j.xml?

The issue is already fixed in the current master but that doesn't help 0.9.0-milestone-1 users :(

Best,

Robert

On Tue, Apr 14, 2015 at 4:49 PM, Stefan Bunk <[hidden email]> wrote:

Hi Flinkers,

I just switched to 0.9.0-milestone-1, and now I get Flink's logging output again in my console (local execution).

I have a log4j.xml under src/main/resources, which says not to log any Flink job progress updates, and which worked fine so far:

[...]

<logger name="org.apache.flink">

                <level value="ERROR" />

                <appender-ref ref="Console" />

</logger>

[...]

See the full file at [1].

However, this file seems to be ignored now. What do I need to change?

Cheers

Stefan

[1] https://gist.github.com/knub/bb1b3ffc6b47e2a1376d

[Stefan Bunk]

Hi Robert,

thanks for the info.

Adding the parameter didn't help. My logging file is found and my logging configuration for my own logging is working (even without the parameter), it's just that the file in the jar seems to be preferred over my file.

Best,

Stefan

On 14 April 2015 at 17:16, Robert Metzger <[hidden email]> wrote:

Hi Stefan,

we made a stupid mistake in the 0.9.0-milestone-1 release by including our log4j.properties into the flink-runtime jar. its also in the fat jar in flink-dist.

Maybe you can pass the name of your log4j file to your application with -Dlog4j.configuration=log4j.xml?

The issue is already fixed in the current master but that doesn't help 0.9.0-milestone-1 users :(

Best,

Robert

On Tue, Apr 14, 2015 at 4:49 PM, Stefan Bunk <[hidden email]> wrote:

Hi Flinkers,

I just switched to 0.9.0-milestone-1, and now I get Flink's logging output again in my console (local execution).

I have a log4j.xml under src/main/resources, which says not to log any Flink job progress updates, and which worked fine so far:

[...]

<logger name="org.apache.flink">

                <level value="ERROR" />

                <appender-ref ref="Console" />

</logger>

[...]

See the full file at [1].

However, this file seems to be ignored now. What do I need to change?

Cheers

Stefan

[1] https://gist.github.com/knub/bb1b3ffc6b47e2a1376d

[rmetzger0]

Hi,

how are you starting Flink? Out of the IDE? Using the scripts?

I just created a new flink project with the milestone version. Just putting your log4j.xml into the resources folder enabled the logging (I've set it to INFO for flink and it worked).

I've used IntelliJ and started the WordCount.java from the quickstart.

I think I need more details about your setup to reproduce the issue.

On Tue, Apr 14, 2015 at 5:52 PM, Stefan Bunk <[hidden email]> wrote:

Hi Robert,

thanks for the info.

Adding the parameter didn't help. My logging file is found and my logging configuration for my own logging is working (even without the parameter), it's just that the file in the jar seems to be preferred over my file.

Best,

Stefan

On 14 April 2015 at 17:16, Robert Metzger <[hidden email]> wrote:

Hi Stefan,

we made a stupid mistake in the 0.9.0-milestone-1 release by including our log4j.properties into the flink-runtime jar. its also in the fat jar in flink-dist.

Maybe you can pass the name of your log4j file to your application with -Dlog4j.configuration=log4j.xml?

The issue is already fixed in the current master but that doesn't help 0.9.0-milestone-1 users :(

Best,

Robert

On Tue, Apr 14, 2015 at 4:49 PM, Stefan Bunk <[hidden email]> wrote:

Hi Flinkers,

I just switched to 0.9.0-milestone-1, and now I get Flink's logging output again in my console (local execution).

I have a log4j.xml under src/main/resources, which says not to log any Flink job progress updates, and which worked fine so far:

[...]

<logger name="org.apache.flink">

                <level value="ERROR" />

                <appender-ref ref="Console" />

</logger>

[...]

See the full file at [1].

However, this file seems to be ignored now. What do I need to change?

Cheers

Stefan

[1] https://gist.github.com/knub/bb1b3ffc6b47e2a1376d

[Stefan Bunk]

Hi,

I am also using IntelliJ, and I am starting directly from the IDE. Local execution.

This is what my logging output looks like: [1].

I am getting my logger via:

val log = org.apache.log4j.Logger.getLogger(getClass)

[1] https://gist.github.com/knub/1c11683601b4eeb5d51b

On 14 April 2015 at 18:47, Robert Metzger <[hidden email]> wrote:

Hi,

how are you starting Flink? Out of the IDE? Using the scripts?

I just created a new flink project with the milestone version. Just putting your log4j.xml into the resources folder enabled the logging (I've set it to INFO for flink and it worked).

I've used IntelliJ and started the WordCount.java from the quickstart.

I think I need more details about your setup to reproduce the issue.

On Tue, Apr 14, 2015 at 5:52 PM, Stefan Bunk <[hidden email]> wrote:

Hi Robert,

thanks for the info.

Adding the parameter didn't help. My logging file is found and my logging configuration for my own logging is working (even without the parameter), it's just that the file in the jar seems to be preferred over my file.

Best,

Stefan

On 14 April 2015 at 17:16, Robert Metzger <[hidden email]> wrote:

Hi Stefan,

we made a stupid mistake in the 0.9.0-milestone-1 release by including our log4j.properties into the flink-runtime jar. its also in the fat jar in flink-dist.

Maybe you can pass the name of your log4j file to your application with -Dlog4j.configuration=log4j.xml?

The issue is already fixed in the current master but that doesn't help 0.9.0-milestone-1 users :(

Best,

Robert

On Tue, Apr 14, 2015 at 4:49 PM, Stefan Bunk <[hidden email]> wrote:

Hi Flinkers,

I just switched to 0.9.0-milestone-1, and now I get Flink's logging output again in my console (local execution).

I have a log4j.xml under src/main/resources, which says not to log any Flink job progress updates, and which worked fine so far:

[...]

<logger name="org.apache.flink">

                <level value="ERROR" />

                <appender-ref ref="Console" />

</logger>

[...]

See the full file at [1].

However, this file seems to be ignored now. What do I need to change?

Cheers

Stefan

[1] https://gist.github.com/knub/bb1b3ffc6b47e2a1376d

[rmetzger0]

Ah, I see.

The issue is this line in the JobClient.scala here: https://github.com/apache/flink/blob/release-0.9.0-milestone-1-rc1/flink-runtime/src/main/scala/org/apache/flink/runtime/client/JobClient.scala#L97

As you can see, its doing sysout logging.

In the current master, this has been reworked: https://github.com/apache/flink/blob/69a400fadd258fe0a1ff0b5670a3611fda4c1cdf/flink-runtime/src/main/java/org/apache/flink/runtime/client/JobClientActor.java#L60

(I'm currently looking up how to control the behavior)

If its possible for you, you can switch to 0.9-SNAPSHOT.

Best,

Robert

On Tue, Apr 14, 2015 at 7:02 PM, Stefan Bunk <[hidden email]> wrote:

Hi,

I am also using IntelliJ, and I am starting directly from the IDE. Local execution.

This is what my logging output looks like: [1].

I am getting my logger via:

val log = org.apache.log4j.Logger.getLogger(getClass)

[1] https://gist.github.com/knub/1c11683601b4eeb5d51b

On 14 April 2015 at 18:47, Robert Metzger <[hidden email]> wrote:

Hi,

how are you starting Flink? Out of the IDE? Using the scripts?

I just created a new flink project with the milestone version. Just putting your log4j.xml into the resources folder enabled the logging (I've set it to INFO for flink and it worked).

I've used IntelliJ and started the WordCount.java from the quickstart.

I think I need more details about your setup to reproduce the issue.

On Tue, Apr 14, 2015 at 5:52 PM, Stefan Bunk <[hidden email]> wrote:

Hi Robert,

thanks for the info.

Adding the parameter didn't help. My logging file is found and my logging configuration for my own logging is working (even without the parameter), it's just that the file in the jar seems to be preferred over my file.

Best,

Stefan

On 14 April 2015 at 17:16, Robert Metzger <[hidden email]> wrote:

Hi Stefan,

we made a stupid mistake in the 0.9.0-milestone-1 release by including our log4j.properties into the flink-runtime jar. its also in the fat jar in flink-dist.

Maybe you can pass the name of your log4j file to your application with -Dlog4j.configuration=log4j.xml?

The issue is already fixed in the current master but that doesn't help 0.9.0-milestone-1 users :(

Best,

Robert

On Tue, Apr 14, 2015 at 4:49 PM, Stefan Bunk <[hidden email]> wrote:

Hi Flinkers,

I just switched to 0.9.0-milestone-1, and now I get Flink's logging output again in my console (local execution).

I have a log4j.xml under src/main/resources, which says not to log any Flink job progress updates, and which worked fine so far:

[...]

<logger name="org.apache.flink">

                <level value="ERROR" />

                <appender-ref ref="Console" />

</logger>

[...]

See the full file at [1].

However, this file seems to be ignored now. What do I need to change?

Cheers

Stefan

[1] https://gist.github.com/knub/bb1b3ffc6b47e2a1376d

[rmetzger0]

You can control the logging behavior from the ExecutionConfig (env.getExecutionConfig()).

There is a method (disableSysoutLogging()) that you can use. (In 0.9-SNAPSHOT only).

Sorry again that you ran into this issue.

On Tue, Apr 14, 2015 at 8:45 PM, Robert Metzger <[hidden email]> wrote:

Ah, I see.

The issue is this line in the JobClient.scala here: https://github.com/apache/flink/blob/release-0.9.0-milestone-1-rc1/flink-runtime/src/main/scala/org/apache/flink/runtime/client/JobClient.scala#L97

As you can see, its doing sysout logging.

In the current master, this has been reworked: https://github.com/apache/flink/blob/69a400fadd258fe0a1ff0b5670a3611fda4c1cdf/flink-runtime/src/main/java/org/apache/flink/runtime/client/JobClientActor.java#L60

(I'm currently looking up how to control the behavior)

If its possible for you, you can switch to 0.9-SNAPSHOT.

Best,

Robert

On Tue, Apr 14, 2015 at 7:02 PM, Stefan Bunk <[hidden email]> wrote:

Hi,

I am also using IntelliJ, and I am starting directly from the IDE. Local execution.

This is what my logging output looks like: [1].

I am getting my logger via:

val log = org.apache.log4j.Logger.getLogger(getClass)

[1] https://gist.github.com/knub/1c11683601b4eeb5d51b

On 14 April 2015 at 18:47, Robert Metzger <[hidden email]> wrote:

Hi,

how are you starting Flink? Out of the IDE? Using the scripts?

I just created a new flink project with the milestone version. Just putting your log4j.xml into the resources folder enabled the logging (I've set it to INFO for flink and it worked).

I've used IntelliJ and started the WordCount.java from the quickstart.

I think I need more details about your setup to reproduce the issue.

On Tue, Apr 14, 2015 at 5:52 PM, Stefan Bunk <[hidden email]> wrote:

Hi Robert,

thanks for the info.

Adding the parameter didn't help. My logging file is found and my logging configuration for my own logging is working (even without the parameter), it's just that the file in the jar seems to be preferred over my file.

Best,

Stefan

On 14 April 2015 at 17:16, Robert Metzger <[hidden email]> wrote:

Hi Stefan,

we made a stupid mistake in the 0.9.0-milestone-1 release by including our log4j.properties into the flink-runtime jar. its also in the fat jar in flink-dist.

Maybe you can pass the name of your log4j file to your application with -Dlog4j.configuration=log4j.xml?

The issue is already fixed in the current master but that doesn't help 0.9.0-milestone-1 users :(

Best,

Robert

On Tue, Apr 14, 2015 at 4:49 PM, Stefan Bunk <[hidden email]> wrote:

Hi Flinkers,

I just switched to 0.9.0-milestone-1, and now I get Flink's logging output again in my console (local execution).

I have a log4j.xml under src/main/resources, which says not to log any Flink job progress updates, and which worked fine so far:

[...]

<logger name="org.apache.flink">

                <level value="ERROR" />

                <appender-ref ref="Console" />

</logger>

[...]

See the full file at [1].

However, this file seems to be ignored now. What do I need to change?

Cheers

Stefan

[1] https://gist.github.com/knub/bb1b3ffc6b47e2a1376d

[Stefan Bunk]

Hi Robert,

this problem persists in the 0.9 release. Using `disableSysoutLogging()` works, but I'd rather configure this in the log4j.xml. Is this possible?

Best,

Stefan

On 14 April 2015 at 20:55, Robert Metzger <[hidden email]> wrote:

You can control the logging behavior from the ExecutionConfig (env.getExecutionConfig()).

There is a method (disableSysoutLogging()) that you can use. (In 0.9-SNAPSHOT only).

Sorry again that you ran into this issue.

On Tue, Apr 14, 2015 at 8:45 PM, Robert Metzger <[hidden email]> wrote:

Ah, I see.

The issue is this line in the JobClient.scala here: https://github.com/apache/flink/blob/release-0.9.0-milestone-1-rc1/flink-runtime/src/main/scala/org/apache/flink/runtime/client/JobClient.scala#L97

As you can see, its doing sysout logging.

In the current master, this has been reworked: https://github.com/apache/flink/blob/69a400fadd258fe0a1ff0b5670a3611fda4c1cdf/flink-runtime/src/main/java/org/apache/flink/runtime/client/JobClientActor.java#L60

(I'm currently looking up how to control the behavior)

If its possible for you, you can switch to 0.9-SNAPSHOT.

Best,

Robert

On Tue, Apr 14, 2015 at 7:02 PM, Stefan Bunk <[hidden email]> wrote:

Hi,

I am also using IntelliJ, and I am starting directly from the IDE. Local execution.

This is what my logging output looks like: [1].

I am getting my logger via:

val log = org.apache.log4j.Logger.getLogger(getClass)

[1] https://gist.github.com/knub/1c11683601b4eeb5d51b

On 14 April 2015 at 18:47, Robert Metzger <[hidden email]> wrote:

Hi,

how are you starting Flink? Out of the IDE? Using the scripts?

I just created a new flink project with the milestone version. Just putting your log4j.xml into the resources folder enabled the logging (I've set it to INFO for flink and it worked).

I've used IntelliJ and started the WordCount.java from the quickstart.

I think I need more details about your setup to reproduce the issue.

On Tue, Apr 14, 2015 at 5:52 PM, Stefan Bunk <[hidden email]> wrote:

Hi Robert,

thanks for the info.

Adding the parameter didn't help. My logging file is found and my logging configuration for my own logging is working (even without the parameter), it's just that the file in the jar seems to be preferred over my file.

Best,

Stefan

On 14 April 2015 at 17:16, Robert Metzger <[hidden email]> wrote:

Hi Stefan,

we made a stupid mistake in the 0.9.0-milestone-1 release by including our log4j.properties into the flink-runtime jar. its also in the fat jar in flink-dist.

Maybe you can pass the name of your log4j file to your application with -Dlog4j.configuration=log4j.xml?

The issue is already fixed in the current master but that doesn't help 0.9.0-milestone-1 users :(

Best,

Robert

On Tue, Apr 14, 2015 at 4:49 PM, Stefan Bunk <[hidden email]> wrote:

Hi Flinkers,

I just switched to 0.9.0-milestone-1, and now I get Flink's logging output again in my console (local execution).

I have a log4j.xml under src/main/resources, which says not to log any Flink job progress updates, and which worked fine so far:

[...]

<logger name="org.apache.flink">

                <level value="ERROR" />

                <appender-ref ref="Console" />

</logger>

[...]

See the full file at [1].

However, this file seems to be ignored now. What do I need to change?

Cheers

Stefan

[1] https://gist.github.com/knub/bb1b3ffc6b47e2a1376d

[Stephan Ewen]

There are two different forms of progress logging:

 - The log4j framework logs everything (as configured in the log4j.properties)

 - Additionally, the client sysout prints the progress reports

If you want the command line to be quiet, you need to disable sysout logging on the execution environment for now.

We are thinking about adding a flag to the CLI frontend to make the console quiet (no progress reports).

Greetings,

Stephan

On Fri, Jun 26, 2015 at 11:47 AM, Stefan Bunk <[hidden email]> wrote:

Hi Robert,

this problem persists in the 0.9 release. Using `disableSysoutLogging()` works, but I'd rather configure this in the log4j.xml. Is this possible?

Best,

Stefan

On 14 April 2015 at 20:55, Robert Metzger <[hidden email]> wrote:

You can control the logging behavior from the ExecutionConfig (env.getExecutionConfig()).

There is a method (disableSysoutLogging()) that you can use. (In 0.9-SNAPSHOT only).

Sorry again that you ran into this issue.

On Tue, Apr 14, 2015 at 8:45 PM, Robert Metzger <[hidden email]> wrote:

Ah, I see.

The issue is this line in the JobClient.scala here: https://github.com/apache/flink/blob/release-0.9.0-milestone-1-rc1/flink-runtime/src/main/scala/org/apache/flink/runtime/client/JobClient.scala#L97

As you can see, its doing sysout logging.

In the current master, this has been reworked: https://github.com/apache/flink/blob/69a400fadd258fe0a1ff0b5670a3611fda4c1cdf/flink-runtime/src/main/java/org/apache/flink/runtime/client/JobClientActor.java#L60

(I'm currently looking up how to control the behavior)

If its possible for you, you can switch to 0.9-SNAPSHOT.

Best,

Robert

On Tue, Apr 14, 2015 at 7:02 PM, Stefan Bunk <[hidden email]> wrote:

Hi,

I am also using IntelliJ, and I am starting directly from the IDE. Local execution.

This is what my logging output looks like: [1].

I am getting my logger via:

val log = org.apache.log4j.Logger.getLogger(getClass)

[1] https://gist.github.com/knub/1c11683601b4eeb5d51b

On 14 April 2015 at 18:47, Robert Metzger <[hidden email]> wrote:

Hi,

how are you starting Flink? Out of the IDE? Using the scripts?

I just created a new flink project with the milestone version. Just putting your log4j.xml into the resources folder enabled the logging (I've set it to INFO for flink and it worked).

I've used IntelliJ and started the WordCount.java from the quickstart.

I think I need more details about your setup to reproduce the issue.

On Tue, Apr 14, 2015 at 5:52 PM, Stefan Bunk <[hidden email]> wrote:

Hi Robert,

thanks for the info.

Adding the parameter didn't help. My logging file is found and my logging configuration for my own logging is working (even without the parameter), it's just that the file in the jar seems to be preferred over my file.

Best,

Stefan

On 14 April 2015 at 17:16, Robert Metzger <[hidden email]> wrote:

Hi Stefan,

we made a stupid mistake in the 0.9.0-milestone-1 release by including our log4j.properties into the flink-runtime jar. its also in the fat jar in flink-dist.

Maybe you can pass the name of your log4j file to your application with -Dlog4j.configuration=log4j.xml?

The issue is already fixed in the current master but that doesn't help 0.9.0-milestone-1 users :(

Best,

Robert

On Tue, Apr 14, 2015 at 4:49 PM, Stefan Bunk <[hidden email]> wrote:

Hi Flinkers,

I just switched to 0.9.0-milestone-1, and now I get Flink's logging output again in my console (local execution).

I have a log4j.xml under src/main/resources, which says not to log any Flink job progress updates, and which worked fine so far:

[...]

<logger name="org.apache.flink">

                <level value="ERROR" />

                <appender-ref ref="Console" />

</logger>

[...]

See the full file at [1].

However, this file seems to be ignored now. What do I need to change?

Cheers

Stefan

[1] https://gist.github.com/knub/bb1b3ffc6b47e2a1376d

[Till Rohrmann]

Hi Stefan,

You can do this if you disableSysoutLogging and change your log4j-cli.properties file to also print to console. There you can then control what is logged to console. However, I think that you have to set disableSysoutLogging in your program.

Cheers,
Till

​

On Fri, Jun 26, 2015 at 11:47 AM, Stefan Bunk <[hidden email]> wrote:

Hi Robert,

this problem persists in the 0.9 release. Using `disableSysoutLogging()` works, but I'd rather configure this in the log4j.xml. Is this possible?

Best,

Stefan

On 14 April 2015 at 20:55, Robert Metzger <[hidden email]> wrote:

You can control the logging behavior from the ExecutionConfig (env.getExecutionConfig()).

There is a method (disableSysoutLogging()) that you can use. (In 0.9-SNAPSHOT only).

Sorry again that you ran into this issue.

On Tue, Apr 14, 2015 at 8:45 PM, Robert Metzger <[hidden email]> wrote:

Ah, I see.

The issue is this line in the JobClient.scala here: https://github.com/apache/flink/blob/release-0.9.0-milestone-1-rc1/flink-runtime/src/main/scala/org/apache/flink/runtime/client/JobClient.scala#L97

As you can see, its doing sysout logging.

In the current master, this has been reworked: https://github.com/apache/flink/blob/69a400fadd258fe0a1ff0b5670a3611fda4c1cdf/flink-runtime/src/main/java/org/apache/flink/runtime/client/JobClientActor.java#L60

(I'm currently looking up how to control the behavior)

If its possible for you, you can switch to 0.9-SNAPSHOT.

Best,

Robert

On Tue, Apr 14, 2015 at 7:02 PM, Stefan Bunk <[hidden email]> wrote:

Hi,

I am also using IntelliJ, and I am starting directly from the IDE. Local execution.

This is what my logging output looks like: [1].

I am getting my logger via:

val log = org.apache.log4j.Logger.getLogger(getClass)

[1] https://gist.github.com/knub/1c11683601b4eeb5d51b

On 14 April 2015 at 18:47, Robert Metzger <[hidden email]> wrote:

Hi,

how are you starting Flink? Out of the IDE? Using the scripts?

I just created a new flink project with the milestone version. Just putting your log4j.xml into the resources folder enabled the logging (I've set it to INFO for flink and it worked).

I've used IntelliJ and started the WordCount.java from the quickstart.

I think I need more details about your setup to reproduce the issue.

On Tue, Apr 14, 2015 at 5:52 PM, Stefan Bunk <[hidden email]> wrote:

Hi Robert,

thanks for the info.

Adding the parameter didn't help. My logging file is found and my logging configuration for my own logging is working (even without the parameter), it's just that the file in the jar seems to be preferred over my file.

Best,

Stefan

On 14 April 2015 at 17:16, Robert Metzger <[hidden email]> wrote:

Hi Stefan,

we made a stupid mistake in the 0.9.0-milestone-1 release by including our log4j.properties into the flink-runtime jar. its also in the fat jar in flink-dist.

Maybe you can pass the name of your log4j file to your application with -Dlog4j.configuration=log4j.xml?

The issue is already fixed in the current master but that doesn't help 0.9.0-milestone-1 users :(

Best,

Robert

On Tue, Apr 14, 2015 at 4:49 PM, Stefan Bunk <[hidden email]> wrote:

Hi Flinkers,

I just switched to 0.9.0-milestone-1, and now I get Flink's logging output again in my console (local execution).

I have a log4j.xml under src/main/resources, which says not to log any Flink job progress updates, and which worked fine so far:

[...]

<logger name="org.apache.flink">

                <level value="ERROR" />

                <appender-ref ref="Console" />

</logger>

[...]

See the full file at [1].

However, this file seems to be ignored now. What do I need to change?

Cheers

Stefan

[1] https://gist.github.com/knub/bb1b3ffc6b47e2a1376d