Configure log4j with XML files

[Gwenhael Pasquiers]

Hi everybody,

 

Could it be possible to have a way to configure log4j with xml files ?

 

I’ve looked into the code and it looks like the properties files names are hardcoded. However we have the need to use xml :

-          We log everything into ELK (Elasticsearch / Logstash / Kibana) using SocketAppender

-          Socket appender is synchronous by default and slow whole app if anything goes wrong with the ELK

-          We usually add an AsyncAppender on top of the SocketAppender, but this sort of configuration is only possible using an XML config file…

 

We’ve already ran into the issue. Everything was almost paused because the ELK was overloaded and extremely slow.

 

B.R.

 

Gwenhaël PASQUIERS

[Till Rohrmann]

Hi Gwenhaël,

as far as I know, there is no direct way to do so. You can either adapt the flink-daemon.sh script in line 68 to use a different configuration or you can test whether the dynamic property -Dlog4j.configurationFile:CONFIG_FILE overrides the -Dlog4j.confguration property. You can set the dynamic property using Flink’s env.java.opts configuration parameter.

Cheers,
Till

​

On Mon, Dec 21, 2015 at 3:34 PM, Gwenhael Pasquiers <[hidden email]> wrote:

Hi everybody,

 

Could it be possible to have a way to configure log4j with xml files ?

 

I’ve looked into the code and it looks like the properties files names are hardcoded. However we have the need to use xml :

-          We log everything into ELK (Elasticsearch / Logstash / Kibana) using SocketAppender

-          Socket appender is synchronous by default and slow whole app if anything goes wrong with the ELK

-          We usually add an AsyncAppender on top of the SocketAppender, but this sort of configuration is only possible using an XML config file…

 

We’ve already ran into the issue. Everything was almost paused because the ELK was overloaded and extremely slow.

 

B.R.

 

Gwenhaël PASQUIERS

[rmetzger0]

as an additional note: Flink is sending all files in the /lib folder to all YARN containers. So you could place the XML file in "/lib" and override the properties.

I think you need to delete the log4j properties from the conf/ directory, then at least on YARN, we'll not set the -Dlog4j.configuration property 

On Mon, Dec 21, 2015 at 3:58 PM, Till Rohrmann <[hidden email]> wrote:

Hi Gwenhaël,

as far as I know, there is no direct way to do so. You can either adapt the flink-daemon.sh script in line 68 to use a different configuration or you can test whether the dynamic property -Dlog4j.configurationFile:CONFIG_FILE overrides the -Dlog4j.confguration property. You can set the dynamic property using Flink’s env.java.opts configuration parameter.

Cheers,
Till

​

On Mon, Dec 21, 2015 at 3:34 PM, Gwenhael Pasquiers <[hidden email]> wrote:

Hi everybody,

 

Could it be possible to have a way to configure log4j with xml files ?

 

I’ve looked into the code and it looks like the properties files names are hardcoded. However we have the need to use xml :

-          We log everything into ELK (Elasticsearch / Logstash / Kibana) using SocketAppender

-          Socket appender is synchronous by default and slow whole app if anything goes wrong with the ELK

-          We usually add an AsyncAppender on top of the SocketAppender, but this sort of configuration is only possible using an XML config file…

 

We’ve already ran into the issue. Everything was almost paused because the ELK was overloaded and extremely slow.

 

B.R.

 

Gwenhaël PASQUIERS