shaded version of legacy kafka connectors

[Gwenhael Pasquiers]

Hi,

 

Before doing it myself I thought it would be better to ask.

We need to consume from kafka 0.8 and produce to kafka 0.10 in a flink app.

I guess there will be classes and package names conflicts for a lot of dependencies of both connectors.

 

The obvious solution it to make a “shaded” version of the kafka 0.8 connector so that it can coexist with the 0.10 version.

 

Does it already exists ?

[Timo Walther]

Hi Gwenhael,

I will loop in Gordon becaue he is more familar with the Kafka connectors. Have you experiences with two versions in the same project?


Am 20/03/17 um 15:57 schrieb Gwenhael Pasquiers:

Hi,

 

Before doing it myself I thought it would be better to ask.

We need to consume from kafka 0.8 and produce to kafka 0.10 in a flink app.

I guess there will be classes and package names conflicts for a lot of dependencies of both connectors.

 

The obvious solution it to make a “shaded” version of the kafka 0.8 connector so that it can coexist with the 0.10 version.

 

Does it already exists ?

[Tzu-Li (Gordon) Tai]

Hi,

There currently isn’t any shaded version Kafka 0.8 connector version available, so yes, you would need to do build that yourself.

I’m not completely sure if there will be any class name clashes, because the Kafka 0.8 API is typically packaged under `kafka.javaapi.*`, while in 0.9 / 0.10 they’re under `org.apache.kafka.*`.

But relocating the `kafka.javaapi.*` classes to a different path is probably the straightforward safe way to go, it shouldn’t have any issues. You shouldn’t need to relocate the Flink Kafka connector classes (under `o.a.f.streaming.connectors.kafka.*`) because there aren’t any name clashes for them between 0.8 and 0.10.

Please feel free to let me know if you bump into any questions with this :-)

Cheers,

Gordon

On March 21, 2017 at 12:43:02 AM, Gwenhael Pasquiers ([hidden email]) wrote:

Hi,

 

Before doing it myself I thought it would be better to ask.

We need to consume from kafka 0.8 and produce to kafka 0.10 in a flink app.

I guess there will be classes and package names conflicts for a lot of dependencies of both connectors.

 

The obvious solution it to make a “shaded” version of the kafka 0.8 connector so that it can coexist with the 0.10 version.

 

Does it already exists ?