(DEPRECATED) Apache Flink User Mailing List archive.

Custom log appender for YARN

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
Gyula Fóra
Reply | Threaded
Open this post in threaded view
♦
♦
|

Custom log appender for YARN

Gyula Fóra
Hi All!

We are trying to configure a custom Kafka log appender for our YARN application and we hit the following problem.

We included the log appender dependency in the fatjar of the application because in YARN that should be part of the system class path.

However when the YARN cluster entrypoints run (YarnJobClusterEntrypoint, YarnTaskExecutorRunner) these only seem to have the contents of lib on their classpath. Does someone have any pointers to how the classpath is configured for running the entrypoints?

The exception is this:
java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
	at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
	at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
	at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:104)

Thanks,
Gyula
Biao Liu
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Custom log appender for YARN

Biao Liu
Hi Gyula,

I guess it should work if you put log appender jar under $flink/lib folder.

There are two different kinds of classloader in your case. One is for Flink framework, the other is for user code. The framework classloader is the parent of user classloader. The parent classloader could not find class from sub-classloader due to the delegation model [1]. In your case, the `ClusterEntrypoint` is in parent classloader, the log appender class in fatjar is in sub-classloader. 

So I think there are two ways to solve this.
1. Put your appender class under $flink/lib folder.
2. Try to avoid using user-defined appender for Flink framework class. (I'm not sure it could be done or not)

You could find more informations in document [2] about Flink classloading strategy.



Thanks,
Biao /'bɪ.aʊ/



On Wed, Jul 31, 2019 at 9:21 PM Gyula Fóra <[hidden email]> wrote:
Hi All!

We are trying to configure a custom Kafka log appender for our YARN application and we hit the following problem.

We included the log appender dependency in the fatjar of the application because in YARN that should be part of the system class path.

However when the YARN cluster entrypoints run (YarnJobClusterEntrypoint, YarnTaskExecutorRunner) these only seem to have the contents of lib on their classpath. Does someone have any pointers to how the classpath is configured for running the entrypoints?

The exception is this:
java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
	at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
	at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
	at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:104)

Thanks,
Gyula
Gyula Fóra
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Custom log appender for YARN

Gyula Fóra
Hi Biao,

Thanks for the input! 
This is basically where we got ourselves as well.

We are trying to avoid putting things in the lib folder so separating the loggers would be great but we don't have a solution for it yet.

Thanks 
Gyula

On Thu, 1 Aug 2019, 07:09 Biao Liu <[hidden email] wrote:
Hi Gyula,

I guess it should work if you put log appender jar under $flink/lib folder.

There are two different kinds of classloader in your case. One is for Flink framework, the other is for user code. The framework classloader is the parent of user classloader. The parent classloader could not find class from sub-classloader due to the delegation model [1]. In your case, the `ClusterEntrypoint` is in parent classloader, the log appender class in fatjar is in sub-classloader. 

So I think there are two ways to solve this.
1. Put your appender class under $flink/lib folder.
2. Try to avoid using user-defined appender for Flink framework class. (I'm not sure it could be done or not)

You could find more informations in document [2] about Flink classloading strategy.



Thanks,
Biao /'bɪ.aʊ/



On Wed, Jul 31, 2019 at 9:21 PM Gyula Fóra <[hidden email]> wrote:
Hi All!

We are trying to configure a custom Kafka log appender for our YARN application and we hit the following problem.

We included the log appender dependency in the fatjar of the application because in YARN that should be part of the system class path.

However when the YARN cluster entrypoints run (YarnJobClusterEntrypoint, YarnTaskExecutorRunner) these only seem to have the contents of lib on their classpath. Does someone have any pointers to how the classpath is configured for running the entrypoints?

The exception is this:
java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
	at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
	at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
	at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:104)

Thanks,
Gyula
Biao Liu
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Custom log appender for YARN

Biao Liu
Hi Gyula,

How about putting the appender class in lib folder, but choosing the appender only for user code through log4j.properties? That way only the user logger would be sent to Kafka. It seems what you want...

Thanks,
Biao /'bɪ.aʊ/



On Thu, Aug 1, 2019 at 2:25 PM Gyula Fóra <[hidden email]> wrote:
Hi Biao,

Thanks for the input! 
This is basically where we got ourselves as well.

We are trying to avoid putting things in the lib folder so separating the loggers would be great but we don't have a solution for it yet.

Thanks 
Gyula

On Thu, 1 Aug 2019, 07:09 Biao Liu <[hidden email] wrote:
Hi Gyula,

I guess it should work if you put log appender jar under $flink/lib folder.

There are two different kinds of classloader in your case. One is for Flink framework, the other is for user code. The framework classloader is the parent of user classloader. The parent classloader could not find class from sub-classloader due to the delegation model [1]. In your case, the `ClusterEntrypoint` is in parent classloader, the log appender class in fatjar is in sub-classloader. 

So I think there are two ways to solve this.
1. Put your appender class under $flink/lib folder.
2. Try to avoid using user-defined appender for Flink framework class. (I'm not sure it could be done or not)

You could find more informations in document [2] about Flink classloading strategy.



Thanks,
Biao /'bɪ.aʊ/



On Wed, Jul 31, 2019 at 9:21 PM Gyula Fóra <[hidden email]> wrote:
Hi All!

We are trying to configure a custom Kafka log appender for our YARN application and we hit the following problem.

We included the log appender dependency in the fatjar of the application because in YARN that should be part of the system class path.

However when the YARN cluster entrypoints run (YarnJobClusterEntrypoint, YarnTaskExecutorRunner) these only seem to have the contents of lib on their classpath. Does someone have any pointers to how the classpath is configured for running the entrypoints?

The exception is this:
java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
	at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
	at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
	at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:104)

Thanks,
Gyula
Free forum by Nabble Edit this page