Fwd: Flink application and curator integration issues

[Liu Tongwei]

Hi all，

I'm using flink 1.1.3. I need to use the curator inside the application to operate zookeeper.

When I use yarn-cluster to submit the application, the following error occurs:

My curator jar package version is 2.8.0, and packaged into the application inside.

It looks like flink import a low version of ZKPaths.

What should I do to solve this problem?

Thanks,

Liu

[Maximilian Michels]

As far as I know we're shading Curator so you shouldn't run into class
conflicts. Have you checked that Curator is included in your jar?

-Max


On Tue, Nov 22, 2016 at 9:30 AM, Liu Tongwei <[hidden email]> wrote:

> Hi all，
>
> I'm using flink 1.1.3. I need to use the curator inside the application to
> operate zookeeper.
> When I use yarn-cluster to submit the application, the following error
> occurs:
>
>
> My curator jar package version is 2.8.0, and packaged into the application
> inside.
> It looks like flink import a low version of ZKPaths.
>
>
> What should I do to solve this problem?
>
> Thanks,
> Liu
>

[Fabian Hueske-2]

This looks rather like a version conflict. If Curator wasn't on the classpath it should be a ClassNotFoundException.

Can you check if any of your jobs dependencies depends on a different Curator version?

Best, Fabian

2016-11-22 12:06 GMT+01:00 Maximilian Michels <[hidden email]>:

As far as I know we're shading Curator so you shouldn't run into class
conflicts. Have you checked that Curator is included in your jar?

-Max


On Tue, Nov 22, 2016 at 9:30 AM, Liu Tongwei <[hidden email]> wrote:
> Hi all，
>
> I'm using flink 1.1.3. I need to use the curator inside the application to
> operate zookeeper.
> When I use yarn-cluster to submit the application, the following error
> occurs:
>
>

> My curator jar package version is 2.8.0, and packaged into the application
> inside.
> It looks like flink import a low version of ZKPaths.
>
>
> What should I do to solve this problem?
>
> Thanks,
> Liu
>

[Stephan Ewen]

Hi Liu!

Did you build Flink from the source yourself?

Maybe you run into the Maven shading problem described here: https://github.com/apache/flink/blob/master/docs/setup/building.md#dependency-shading

Best,

Stephan

On Tue, Nov 22, 2016 at 8:03 PM, Fabian Hueske <[hidden email]> wrote:

This looks rather like a version conflict. If Curator wasn't on the classpath it should be a ClassNotFoundException.

Can you check if any of your jobs dependencies depends on a different Curator version?

Best, Fabian

2016-11-22 12:06 GMT+01:00 Maximilian Michels <[hidden email]>:

As far as I know we're shading Curator so you shouldn't run into class
conflicts. Have you checked that Curator is included in your jar?

-Max


On Tue, Nov 22, 2016 at 9:30 AM, Liu Tongwei <[hidden email]> wrote:
> Hi all，
>
> I'm using flink 1.1.3. I need to use the curator inside the application to
> operate zookeeper.
> When I use yarn-cluster to submit the application, the following error
> occurs:
>
>

> My curator jar package version is 2.8.0, and packaged into the application
> inside.
> It looks like flink import a low version of ZKPaths.
>
>
> What should I do to solve this problem?
>
> Thanks,
> Liu
>

[Liu Tongwei]

Thanks Stephan,what you said really solved the problem.
Previously I build Flink against a vendor specific Hadoop version and my Maven version is 3.3.9.
 
Thanks,
Liu