Custom metrics reporter classloading problem

[Gyula Fóra]

Hi all,

I have ran into the following problem and I want to double check wether this is intended behaviour.

I have a custom metrics reporter that pushes things to Kafka (so it creates a KafkaProducer in the open method etc.etc.) for my streaming job.

Naturally as my Flink job consumes from Kafka so it has the kafka connector dependencies I set the Kafka dependencies to provided in my metric reporter project and I put the built kafkaReporter.jar into the Flink lib. However it seems that the metrics reporter is instantiated without the user code classes since I get a NoClassdefFound error for KafkaProducer even though my streaming job starts successfully reading/writing kafka.

Any ideas why this happens and how to solve it? I am slightly against putting the kafka dependencies twice on the classpath as it has only caused problems in the past...

Gyula

[Chesnay Schepler]

Reporters do not have access to libraries provided with user-jars.
They are instantiated when JM/TM starts, i.e. before any user-code is
even accessible.

My recommendation would be to either put the kafka dependencies in the
/lib folder or try to relocate the kafka code in the reporter.

On 11.07.2018 14:59, Gyula Fóra wrote:

> Hi all,
>
> I have ran into the following problem and I want to double check
> wether this is intended behaviour.
>
> I have a custom metrics reporter that pushes things to Kafka (so it
> creates a KafkaProducer in the open method etc.etc.) for my streaming job.
>
> Naturally as my Flink job consumes from Kafka so it has the kafka
> connector dependencies I set the Kafka dependencies to provided in my
> metric reporter project and I put the built kafkaReporter.jar into the
> Flink lib. However it seems that the metrics reporter is instantiated
> without the user code classes since I get a NoClassdefFound error for
> KafkaProducer even though my streaming job starts successfully
> reading/writing kafka.
>
> Any ideas why this happens and how to solve it? I am slightly against
> putting the kafka dependencies twice on the classpath as it has only
> caused problems in the past...
>
> Gyula

[Gyula Fóra]

Thanks for the explanation, that makes sense.

For some reason I thought that in Yarn all stuff goes into the classpath.

Gy

Chesnay Schepler <[hidden email]> ezt írta (időpont: 2018. júl. 11., Sze, 15:16):

Reporters do not have access to libraries provided with user-jars.
They are instantiated when JM/TM starts, i.e. before any user-code is
even accessible.

My recommendation would be to either put the kafka dependencies in the
/lib folder or try to relocate the kafka code in the reporter.

On 11.07.2018 14:59, Gyula Fóra wrote:
> Hi all,
>
> I have ran into the following problem and I want to double check
> wether this is intended behaviour.
>
> I have a custom metrics reporter that pushes things to Kafka (so it
> creates a KafkaProducer in the open method etc.etc.) for my streaming job.
>
> Naturally as my Flink job consumes from Kafka so it has the kafka
> connector dependencies I set the Kafka dependencies to provided in my
> metric reporter project and I put the built kafkaReporter.jar into the
> Flink lib. However it seems that the metrics reporter is instantiated
> without the user code classes since I get a NoClassdefFound error for
> KafkaProducer even though my streaming job starts successfully
> reading/writing kafka.
>
> Any ideas why this happens and how to solve it? I am slightly against
> putting the kafka dependencies twice on the classpath as it has only
> caused problems in the past...
>
> Gyula