Log4J

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Log4J

Chet Masterson
Is there a way to reload a log4j.properties file without stopping and starting the job server?
Reply | Threaded
Open this post in threaded view
|

Re: Log4J

Stephan Ewen
Hi!

The bundled log4j version (1.x) does not support that.

But you can replace the logging jars with those of a different framework (like log4j 2.x), which supports changing the configuration without stopping the application.
You don't need to rebuild flink, simply replace two jars in the "lib" folder (and update the config file, because log4j 2.x has a different config format).

This guide shows how to swap log4j 1.x for logback, and you should be able to swap in log4j 2.x in the exact same way.



On Thu, Feb 16, 2017 at 5:20 AM, Chet Masterson <[hidden email]> wrote:
Is there a way to reload a log4j.properties file without stopping and starting the job server?

Reply | Threaded
Open this post in threaded view
|

Re: Log4J

rmetzger0
I've also (successfully) tried running Flink with log4j2 to connect it to greylog2. If I remember correctly, the biggest problem was "injecting" the log4j2 properties file into the classpath (when running Flink on YARN).

Maybe you need to put the file into the lib/ folder, so that it is shipped to all the nodes, and then loaded from the classpath (there is a special name in the log4j2 documentation. If you use that name, it'll be loaded from the classloader)

If you are running in standalone mode, you can just modify the scripts to point the JVMs to the right config file.

On Thu, Feb 16, 2017 at 11:54 AM, Stephan Ewen <[hidden email]> wrote:
Hi!

The bundled log4j version (1.x) does not support that.

But you can replace the logging jars with those of a different framework (like log4j 2.x), which supports changing the configuration without stopping the application.
You don't need to rebuild flink, simply replace two jars in the "lib" folder (and update the config file, because log4j 2.x has a different config format).

This guide shows how to swap log4j 1.x for logback, and you should be able to swap in log4j 2.x in the exact same way.



On Thu, Feb 16, 2017 at 5:20 AM, Chet Masterson <[hidden email]> wrote:
Is there a way to reload a log4j.properties file without stopping and starting the job server?


Reply | Threaded
Open this post in threaded view
|

Re: Log4J

Chet Masterson
In reply to this post by Stephan Ewen
I read through the link you provided, Stephan. However, I am still confused. The instructions mention specific jar files for Logback, I am not sure which of the log4j 2.x jars I need to put in the the flink /lib directory. I tried various combinations of log4j-1.2-api-2.8.jar, log4j-slf4j-impl-2.8.jar, log4j-to-slf4j-2.8.jar, and renamed the stock log4j-1.2.17.jar and slf4j-log4j12-1.7.7.jar, but then the job manager would not start, and threw a 'NoClassDefFoundError: org/apache/logging/log4j/LogManager'. And this is without deploying my job out there, so I don't think any of the "Use Logback when running Flink out of the IDE / from a Java application" section instructions are relevant.
 
Can someone be more specific how to do this? If I get it to work, I'll be happy to formally document it in whatever format would help the project out long term.
 
Thanks!
 
 
16.02.2017, 05:54, "Stephan Ewen" <[hidden email]>:
Hi!
 
The bundled log4j version (1.x) does not support that.
 
But you can replace the logging jars with those of a different framework (like log4j 2.x), which supports changing the configuration without stopping the application.
You don't need to rebuild flink, simply replace two jars in the "lib" folder (and update the config file, because log4j 2.x has a different config format).
 
This guide shows how to swap log4j 1.x for logback, and you should be able to swap in log4j 2.x in the exact same way.
 
 
 
On Thu, Feb 16, 2017 at 5:20 AM, Chet Masterson <[hidden email]> wrote:
Is there a way to reload a log4j.properties file without stopping and starting the job server?
Reply | Threaded
Open this post in threaded view
|

Re: Log4J

rmetzger0
Hi Chet,

These are the files I have in my lib/ folder with the working log4j2 integration:

-rw-r--r--  1 robert robert 79966937 Oct 10 13:49 flink-dist_2.10-1.1.3.jar
-rw-r--r--  1 robert robert    90883 Dec  9 20:13 flink-python_2.10-1.1.3.jar
-rw-r--r--  1 robert robert    60547 Dec  9 18:45 log4j-1.2-api-2.7.jar
-rw-rw-r--  1 robert robert  1638598 Oct 22 16:08 log4j2-gelf-1.3.1-shaded.jar
-rw-rw-r--  1 robert robert     1056 Dec  9 20:12 log4j2.properties
-rw-r--r--  1 robert robert   219001 Dec  9 18:45 log4j-api-2.7.jar
-rw-r--r--  1 robert robert  1296865 Dec  9 18:45 log4j-core-2.7.jar
-rw-r--r--  1 robert robert    22918 Dec  9 18:46 log4j-slf4j-impl-2.7.jar

You don't need the "log4j2-gelf-1.3.1-shaded.jar", that's a GELF appender for Greylog2.

On Mon, Feb 20, 2017 at 5:41 AM, Chet Masterson <[hidden email]> wrote:
I read through the link you provided, Stephan. However, I am still confused. The instructions mention specific jar files for Logback, I am not sure which of the log4j 2.x jars I need to put in the the flink /lib directory. I tried various combinations of log4j-1.2-api-2.8.jar, log4j-slf4j-impl-2.8.jar, log4j-to-slf4j-2.8.jar, and renamed the stock log4j-1.2.17.jar and slf4j-log4j12-1.7.7.jar, but then the job manager would not start, and threw a 'NoClassDefFoundError: org/apache/logging/log4j/LogManager'. And this is without deploying my job out there, so I don't think any of the "Use Logback when running Flink out of the IDE / from a Java application" section instructions are relevant.
 
Can someone be more specific how to do this? If I get it to work, I'll be happy to formally document it in whatever format would help the project out long term.
 
Thanks!
 
 
16.02.2017, 05:54, "Stephan Ewen" <[hidden email]>:
Hi!
 
The bundled log4j version (1.x) does not support that.
 
But you can replace the logging jars with those of a different framework (like log4j 2.x), which supports changing the configuration without stopping the application.
You don't need to rebuild flink, simply replace two jars in the "lib" folder (and update the config file, because log4j 2.x has a different config format).
 
This guide shows how to swap log4j 1.x for logback, and you should be able to swap in log4j 2.x in the exact same way.
 
 
 
On Thu, Feb 16, 2017 at 5:20 AM, Chet Masterson <[hidden email]> wrote:
Is there a way to reload a log4j.properties file without stopping and starting the job server?

Reply | Threaded
Open this post in threaded view
|

Re: Log4J

Stephan Ewen
How about adding this to the "logging" docs - a section on how to run log4j2

On Mon, Feb 20, 2017 at 8:50 AM, Robert Metzger <[hidden email]> wrote:
Hi Chet,

These are the files I have in my lib/ folder with the working log4j2 integration:

-rw-r--r--  1 robert robert 79966937 Oct 10 13:49 flink-dist_2.10-1.1.3.jar
-rw-r--r--  1 robert robert    90883 Dec  9 20:13 flink-python_2.10-1.1.3.jar
-rw-r--r--  1 robert robert    60547 Dec  9 18:45 log4j-1.2-api-2.7.jar
-rw-rw-r--  1 robert robert  1638598 Oct 22 16:08 log4j2-gelf-1.3.1-shaded.jar
-rw-rw-r--  1 robert robert     1056 Dec  9 20:12 log4j2.properties
-rw-r--r--  1 robert robert   219001 Dec  9 18:45 log4j-api-2.7.jar
-rw-r--r--  1 robert robert  1296865 Dec  9 18:45 log4j-core-2.7.jar
-rw-r--r--  1 robert robert    22918 Dec  9 18:46 log4j-slf4j-impl-2.7.jar

You don't need the "log4j2-gelf-1.3.1-shaded.jar", that's a GELF appender for Greylog2.

On Mon, Feb 20, 2017 at 5:41 AM, Chet Masterson <[hidden email]> wrote:
I read through the link you provided, Stephan. However, I am still confused. The instructions mention specific jar files for Logback, I am not sure which of the log4j 2.x jars I need to put in the the flink /lib directory. I tried various combinations of log4j-1.2-api-2.8.jar, log4j-slf4j-impl-2.8.jar, log4j-to-slf4j-2.8.jar, and renamed the stock log4j-1.2.17.jar and slf4j-log4j12-1.7.7.jar, but then the job manager would not start, and threw a 'NoClassDefFoundError: org/apache/logging/log4j/LogManager'. And this is without deploying my job out there, so I don't think any of the "Use Logback when running Flink out of the IDE / from a Java application" section instructions are relevant.
 
Can someone be more specific how to do this? If I get it to work, I'll be happy to formally document it in whatever format would help the project out long term.
 
Thanks!
 
 
16.02.2017, 05:54, "Stephan Ewen" <[hidden email]>:
Hi!
 
The bundled log4j version (1.x) does not support that.
 
But you can replace the logging jars with those of a different framework (like log4j 2.x), which supports changing the configuration without stopping the application.
You don't need to rebuild flink, simply replace two jars in the "lib" folder (and update the config file, because log4j 2.x has a different config format).
 
This guide shows how to swap log4j 1.x for logback, and you should be able to swap in log4j 2.x in the exact same way.
 
 
 
On Thu, Feb 16, 2017 at 5:20 AM, Chet Masterson <[hidden email]> wrote:
Is there a way to reload a log4j.properties file without stopping and starting the job server?


Reply | Threaded
Open this post in threaded view
|

Re: Log4J

rmetzger0
Absolutely agreed. I have such a task on my todo list and I hope to find time to address this soon.

On Mon, Feb 20, 2017 at 8:08 PM, Stephan Ewen <[hidden email]> wrote:
How about adding this to the "logging" docs - a section on how to run log4j2

On Mon, Feb 20, 2017 at 8:50 AM, Robert Metzger <[hidden email]> wrote:
Hi Chet,

These are the files I have in my lib/ folder with the working log4j2 integration:

-rw-r--r--  1 robert robert 79966937 Oct 10 13:49 flink-dist_2.10-1.1.3.jar
-rw-r--r--  1 robert robert    90883 Dec  9 20:13 flink-python_2.10-1.1.3.jar
-rw-r--r--  1 robert robert    60547 Dec  9 18:45 log4j-1.2-api-2.7.jar
-rw-rw-r--  1 robert robert  1638598 Oct 22 16:08 log4j2-gelf-1.3.1-shaded.jar
-rw-rw-r--  1 robert robert     1056 Dec  9 20:12 log4j2.properties
-rw-r--r--  1 robert robert   219001 Dec  9 18:45 log4j-api-2.7.jar
-rw-r--r--  1 robert robert  1296865 Dec  9 18:45 log4j-core-2.7.jar
-rw-r--r--  1 robert robert    22918 Dec  9 18:46 log4j-slf4j-impl-2.7.jar

You don't need the "log4j2-gelf-1.3.1-shaded.jar", that's a GELF appender for Greylog2.

On Mon, Feb 20, 2017 at 5:41 AM, Chet Masterson <[hidden email]> wrote:
I read through the link you provided, Stephan. However, I am still confused. The instructions mention specific jar files for Logback, I am not sure which of the log4j 2.x jars I need to put in the the flink /lib directory. I tried various combinations of log4j-1.2-api-2.8.jar, log4j-slf4j-impl-2.8.jar, log4j-to-slf4j-2.8.jar, and renamed the stock log4j-1.2.17.jar and slf4j-log4j12-1.7.7.jar, but then the job manager would not start, and threw a 'NoClassDefFoundError: org/apache/logging/log4j/LogManager'. And this is without deploying my job out there, so I don't think any of the "Use Logback when running Flink out of the IDE / from a Java application" section instructions are relevant.
 
Can someone be more specific how to do this? If I get it to work, I'll be happy to formally document it in whatever format would help the project out long term.
 
Thanks!
 
 
16.02.2017, 05:54, "Stephan Ewen" <[hidden email]>:
Hi!
 
The bundled log4j version (1.x) does not support that.
 
But you can replace the logging jars with those of a different framework (like log4j 2.x), which supports changing the configuration without stopping the application.
You don't need to rebuild flink, simply replace two jars in the "lib" folder (and update the config file, because log4j 2.x has a different config format).
 
This guide shows how to swap log4j 1.x for logback, and you should be able to swap in log4j 2.x in the exact same way.
 
 
 
On Thu, Feb 16, 2017 at 5:20 AM, Chet Masterson <[hidden email]> wrote:
Is there a way to reload a log4j.properties file without stopping and starting the job server?