Possible conflict between in Flink connectors

[Federico D'Ambrosio]

Hello everyone, I'm new to Flink and am encountering a nasty problem while trying to submit a streaming Flink Job. I'll try to explain it as thoroughly as possible.

Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as per documentation and I submit the job using yarn-session.sh and then flink run.

Getting into more details, the Flink Job is a fat jar built with sbt assembly and the following dependencies:

libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion % "provided"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"


assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

inside the flink lib folder I have the following Jars:

flink-connector-filesystem_2.10-1.3.2.jar 

flink-python_2.10-1.3.2.jar
flink-connector-kinesis_2.10-1.3.2.jar    

flink-shaded-hadoop2-uber-1.3.2.jar
flink-dist_2.10-1.3.2.jar                 

log4j-1.2.17.jar
flink-hbase_2.10-1.3.2.jar               

slf4j-log4j12-1.7.7.jar

Right now, the issue is that when I flink run the job, I get NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it being inside the fat jar. So, I tried putting both hbase-common.jar and hbase-client.jars inside the lib folder, getting as a result another NoClassDefFound, only, this time, for com/google/common/collect/ListMultimap. Now, I noticed that flink-connector-kinesis and all the hbase packages (both hbase-common and flink-hbase deps) have a different version of guava as dependency: the first one needs version 18.0, while the second one needs 12.0.1 version. I, then, tried to enable the shading of guava packages with the following:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
)

Only to no avail, getting the exact same error as before. And I really don't know how to submit the job without pouring every single dependency inside the lib folder. I thought it could be a compatibility issue with the YARN cluster, but I tried launching the job locally only to get the same error.

As an additional info, I get the following warnings if I don't specify flink packages as provided:

[warn]     * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over 2.0.1
[warn]         +- org.typelevel:macro-compat_2.10:1.1.1              (depends on 2.1.0)
[warn]         +- org.apache.flink:flink-scala_2.10:1.3.2            (depends on 2.0.1)
[warn]
[warn]     * org.mortbay.jetty:jetty-util:6.1.26 is selected over 6.1.26.hwx
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 6.1.26)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2        (depends on 6.1.26.hwx)
[warn]
[warn]     * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
[warn]         +- com.data-artisans:flakka-remote_2.10:2.3-custom    (depends on 3.8.0.Final)
[warn]         +- org.apache.zookeeper:zookeeper:3.4.6               (depends on 3.7.0.Final)
[warn]
[warn]     * com.google.guava:guava:18.0 is selected over 12.0.1
[warn]         +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2 (depends on 18.0)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0             (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 12.0.1)
[warn]
[warn]     * io.netty:netty-all:4.0.27.Final is selected over 4.0.23.Final
[warn]         +- org.apache.flink:flink-runtime_2.10:1.3.2          (depends on 4.0.27.Final)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2        (depends on 4.0.27.Final)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 4.0.23.Final)
[warn]
[warn]     * junit:junit:4.12 is selected over 3.8.1
[warn]         +- org.apache.hbase:hbase-protocol:1.3.0              (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-annotations:1.3.0           (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0             (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 4.12)
[warn]         +- jline:jline:0.9.94                                 (depends on 3.8.1)

And here I am asking for help here.

Thank you very much for your attention,

Kind regards,

Federico D'Ambrosio

[Ted Yu]

If Guava 18.0 is used to build hbase 1.3, there would be compilation errors such as the following:

[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[271,25] error: cannot find symbol

[ERROR]   symbol:   method stopAndWait()

[ERROR]   location: variable replicationEndpoint of type ReplicationEndpoint

[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[281,47] error: cannot find symbol

[ERROR]   symbol:   method start()

[ERROR]   location: variable replicationEndpoint of type ReplicationEndpoint

Maybe you can shade the guava dependency in hbase 1.3

In the upcoming hbase 2.0 release, third party dependencies such as guava and netty are shaded. Meaning there wouldn't be such conflict with other components you may use.

On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio <[hidden email]> wrote:

Hello everyone, I'm new to Flink and am encountering a nasty problem while trying to submit a streaming Flink Job. I'll try to explain it as thoroughly as possible.

Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as per documentation and I submit the job using yarn-session.sh and then flink run.

Getting into more details, the Flink Job is a fat jar built with sbt assembly and the following dependencies:

libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion % "provided"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"


assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

inside the flink lib folder I have the following Jars:

flink-connector-filesystem_2.10-1.3.2.jar 

flink-python_2.10-1.3.2.jar
flink-connector-kinesis_2.10-1.3.2.jar    

flink-shaded-hadoop2-uber-1.3.2.jar
flink-dist_2.10-1.3.2.jar                 

log4j-1.2.17.jar
flink-hbase_2.10-1.3.2.jar               

slf4j-log4j12-1.7.7.jar

Right now, the issue is that when I flink run the job, I get NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it being inside the fat jar. So, I tried putting both hbase-common.jar and hbase-client.jars inside the lib folder, getting as a result another NoClassDefFound, only, this time, for com/google/common/collect/ListMultimap. Now, I noticed that flink-connector-kinesis and all the hbase packages (both hbase-common and flink-hbase deps) have a different version of guava as dependency: the first one needs version 18.0, while the second one needs 12.0.1 version. I, then, tried to enable the shading of guava packages with the following:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
)

Only to no avail, getting the exact same error as before. And I really don't know how to submit the job without pouring every single dependency inside the lib folder. I thought it could be a compatibility issue with the YARN cluster, but I tried launching the job locally only to get the same error.

As an additional info, I get the following warnings if I don't specify flink packages as provided:

[warn]     * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over 2.0.1
[warn]         +- org.typelevel:macro-compat_2.10:1.1.1              (depends on 2.1.0)
[warn]         +- org.apache.flink:flink-scala_2.10:1.3.2            (depends on 2.0.1)
[warn]
[warn]     * org.mortbay.jetty:jetty-util:6.1.26 is selected over 6.1.26.hwx
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 6.1.26)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2        (depends on 6.1.26.hwx)
[warn]
[warn]     * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
[warn]         +- com.data-artisans:flakka-remote_2.10:2.3-custom    (depends on 3.8.0.Final)
[warn]         +- org.apache.zookeeper:zookeeper:3.4.6               (depends on 3.7.0.Final)
[warn]
[warn]     * com.google.guava:guava:18.0 is selected over 12.0.1
[warn]         +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2 (depends on 18.0)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0             (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 12.0.1)
[warn]
[warn]     * io.netty:netty-all:4.0.27.Final is selected over 4.0.23.Final
[warn]         +- org.apache.flink:flink-runtime_2.10:1.3.2          (depends on 4.0.27.Final)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2        (depends on 4.0.27.Final)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 4.0.23.Final)
[warn]
[warn]     * junit:junit:4.12 is selected over 3.8.1
[warn]         +- org.apache.hbase:hbase-protocol:1.3.0              (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-annotations:1.3.0           (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0             (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 4.12)
[warn]         +- jline:jline:0.9.94                                 (depends on 3.8.1)

And here I am asking for help here.

Thank you very much for your attention,

Kind regards,

Federico D'Ambrosio

[rmetzger0]

Hi,

why do you need to add hbase-common as a separate dependency? Doesn't the "flink-hbase" dependency transitively pull in hbase?

On Fri, Aug 25, 2017 at 6:35 PM, Ted Yu <[hidden email]> wrote:

If Guava 18.0 is used to build hbase 1.3, there would be compilation errors such as the following:

[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[271,25] error: cannot find symbol

[ERROR]   symbol:   method stopAndWait()

[ERROR]   location: variable replicationEndpoint of type ReplicationEndpoint

[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[281,47] error: cannot find symbol

[ERROR]   symbol:   method start()

[ERROR]   location: variable replicationEndpoint of type ReplicationEndpoint

Maybe you can shade the guava dependency in hbase 1.3

In the upcoming hbase 2.0 release, third party dependencies such as guava and netty are shaded. Meaning there wouldn't be such conflict with other components you may use.

On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio <[hidden email]> wrote:

Hello everyone, I'm new to Flink and am encountering a nasty problem while trying to submit a streaming Flink Job. I'll try to explain it as thoroughly as possible.

Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as per documentation and I submit the job using yarn-session.sh and then flink run.

Getting into more details, the Flink Job is a fat jar built with sbt assembly and the following dependencies:

libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion % "provided"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"


assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

inside the flink lib folder I have the following Jars:

flink-connector-filesystem_2.10-1.3.2.jar 

flink-python_2.10-1.3.2.jar
flink-connector-kinesis_2.10-1.3.2.jar    

flink-shaded-hadoop2-uber-1.3.2.jar
flink-dist_2.10-1.3.2.jar                 

log4j-1.2.17.jar
flink-hbase_2.10-1.3.2.jar               

slf4j-log4j12-1.7.7.jar

Right now, the issue is that when I flink run the job, I get NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it being inside the fat jar. So, I tried putting both hbase-common.jar and hbase-client.jars inside the lib folder, getting as a result another NoClassDefFound, only, this time, for com/google/common/collect/ListMultimap. Now, I noticed that flink-connector-kinesis and all the hbase packages (both hbase-common and flink-hbase deps) have a different version of guava as dependency: the first one needs version 18.0, while the second one needs 12.0.1 version. I, then, tried to enable the shading of guava packages with the following:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
)

Only to no avail, getting the exact same error as before. And I really don't know how to submit the job without pouring every single dependency inside the lib folder. I thought it could be a compatibility issue with the YARN cluster, but I tried launching the job locally only to get the same error.

As an additional info, I get the following warnings if I don't specify flink packages as provided:

[warn]     * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over 2.0.1
[warn]         +- org.typelevel:macro-compat_2.10:1.1.1              (depends on 2.1.0)
[warn]         +- org.apache.flink:flink-scala_2.10:1.3.2            (depends on 2.0.1)
[warn]
[warn]     * org.mortbay.jetty:jetty-util:6.1.26 is selected over 6.1.26.hwx
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 6.1.26)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2        (depends on 6.1.26.hwx)
[warn]
[warn]     * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
[warn]         +- com.data-artisans:flakka-remote_2.10:2.3-custom    (depends on 3.8.0.Final)
[warn]         +- org.apache.zookeeper:zookeeper:3.4.6               (depends on 3.7.0.Final)
[warn]
[warn]     * com.google.guava:guava:18.0 is selected over 12.0.1
[warn]         +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2 (depends on 18.0)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0             (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 12.0.1)
[warn]
[warn]     * io.netty:netty-all:4.0.27.Final is selected over 4.0.23.Final
[warn]         +- org.apache.flink:flink-runtime_2.10:1.3.2          (depends on 4.0.27.Final)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2        (depends on 4.0.27.Final)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 4.0.23.Final)
[warn]
[warn]     * junit:junit:4.12 is selected over 3.8.1
[warn]         +- org.apache.hbase:hbase-protocol:1.3.0              (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-annotations:1.3.0           (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0             (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 4.12)
[warn]         +- jline:jline:0.9.94                                 (depends on 3.8.1)

And here I am asking for help here.

Thank you very much for your attention,

Kind regards,

Federico D'Ambrosio

[Ted Yu]

Looking at dependencies for flink-hbase, we have :

[INFO] +- org.apache.hbase:hbase-server:jar:1.3.1:compile

[INFO] |  +- org.apache.hbase:hbase-common:jar:1.3.1:compile

[INFO] |  +- org.apache.hbase:hbase-protocol:jar:1.3.1:compile

[INFO] |  +- org.apache.hbase:hbase-procedure:jar:1.3.1:compile

[INFO] |  |  \- org.apache.hbase:hbase-common:jar:tests:1.3.1:compile

[INFO] |  +- org.apache.hbase:hbase-client:jar:1.3.1:compile

On Fri, Aug 25, 2017 at 2:03 PM, Robert Metzger <[hidden email]> wrote:

Hi,

why do you need to add hbase-common as a separate dependency? Doesn't the "flink-hbase" dependency transitively pull in hbase?

On Fri, Aug 25, 2017 at 6:35 PM, Ted Yu <[hidden email]> wrote:

If Guava 18.0 is used to build hbase 1.3, there would be compilation errors such as the following:

[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[271,25] error: cannot find symbol

[ERROR]   symbol:   method stopAndWait()

[ERROR]   location: variable replicationEndpoint of type ReplicationEndpoint

[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[281,47] error: cannot find symbol

[ERROR]   symbol:   method start()

[ERROR]   location: variable replicationEndpoint of type ReplicationEndpoint

Maybe you can shade the guava dependency in hbase 1.3

In the upcoming hbase 2.0 release, third party dependencies such as guava and netty are shaded. Meaning there wouldn't be such conflict with other components you may use.

On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio <[hidden email]> wrote:

Hello everyone, I'm new to Flink and am encountering a nasty problem while trying to submit a streaming Flink Job. I'll try to explain it as thoroughly as possible.

Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as per documentation and I submit the job using yarn-session.sh and then flink run.

Getting into more details, the Flink Job is a fat jar built with sbt assembly and the following dependencies:

libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion % "provided"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"


assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

inside the flink lib folder I have the following Jars:

flink-connector-filesystem_2.10-1.3.2.jar 

flink-python_2.10-1.3.2.jar
flink-connector-kinesis_2.10-1.3.2.jar    

flink-shaded-hadoop2-uber-1.3.2.jar
flink-dist_2.10-1.3.2.jar                 

log4j-1.2.17.jar
flink-hbase_2.10-1.3.2.jar               

slf4j-log4j12-1.7.7.jar

Right now, the issue is that when I flink run the job, I get NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it being inside the fat jar. So, I tried putting both hbase-common.jar and hbase-client.jars inside the lib folder, getting as a result another NoClassDefFound, only, this time, for com/google/common/collect/ListMultimap. Now, I noticed that flink-connector-kinesis and all the hbase packages (both hbase-common and flink-hbase deps) have a different version of guava as dependency: the first one needs version 18.0, while the second one needs 12.0.1 version. I, then, tried to enable the shading of guava packages with the following:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
)

Only to no avail, getting the exact same error as before. And I really don't know how to submit the job without pouring every single dependency inside the lib folder. I thought it could be a compatibility issue with the YARN cluster, but I tried launching the job locally only to get the same error.

As an additional info, I get the following warnings if I don't specify flink packages as provided:

[warn]     * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over 2.0.1
[warn]         +- org.typelevel:macro-compat_2.10:1.1.1              (depends on 2.1.0)
[warn]         +- org.apache.flink:flink-scala_2.10:1.3.2            (depends on 2.0.1)
[warn]
[warn]     * org.mortbay.jetty:jetty-util:6.1.26 is selected over 6.1.26.hwx
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 6.1.26)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2        (depends on 6.1.26.hwx)
[warn]
[warn]     * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
[warn]         +- com.data-artisans:flakka-remote_2.10:2.3-custom    (depends on 3.8.0.Final)
[warn]         +- org.apache.zookeeper:zookeeper:3.4.6               (depends on 3.7.0.Final)
[warn]
[warn]     * com.google.guava:guava:18.0 is selected over 12.0.1
[warn]         +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2 (depends on 18.0)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0             (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 12.0.1)
[warn]
[warn]     * io.netty:netty-all:4.0.27.Final is selected over 4.0.23.Final
[warn]         +- org.apache.flink:flink-runtime_2.10:1.3.2          (depends on 4.0.27.Final)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2        (depends on 4.0.27.Final)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 4.0.23.Final)
[warn]
[warn]     * junit:junit:4.12 is selected over 3.8.1
[warn]         +- org.apache.hbase:hbase-protocol:1.3.0              (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-annotations:1.3.0           (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0             (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 4.12)
[warn]         +- jline:jline:0.9.94                                 (depends on 3.8.1)

And here I am asking for help here.

Thank you very much for your attention,

Kind regards,

Federico D'Ambrosio

[Federico D'Ambrosio]

Hi,

At the beginning, I was wondering myself that too, and I don't know why hbase-common wasn''t being downloaded and included, so I added it explicitly.

I was in the process to write that maybe I've solved this weird issue: apparently the shading worked and the ClassDefNotFound issue was caused by the missing hbase-client jar inside the fat jar, even though there should have been! So, I added again the explicit dependency for it in the build.sbt and now it's working.

2017-08-25 23:03 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

why do you need to add hbase-common as a separate dependency? Doesn't the "flink-hbase" dependency transitively pull in hbase?

On Fri, Aug 25, 2017 at 6:35 PM, Ted Yu <[hidden email]> wrote:

If Guava 18.0 is used to build hbase 1.3, there would be compilation errors such as the following:

[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[271,25] error: cannot find symbol

[ERROR]   symbol:   method stopAndWait()

[ERROR]   location: variable replicationEndpoint of type ReplicationEndpoint

[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[281,47] error: cannot find symbol

[ERROR]   symbol:   method start()

[ERROR]   location: variable replicationEndpoint of type ReplicationEndpoint

Maybe you can shade the guava dependency in hbase 1.3

In the upcoming hbase 2.0 release, third party dependencies such as guava and netty are shaded. Meaning there wouldn't be such conflict with other components you may use.

On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio <[hidden email]> wrote:

Hello everyone, I'm new to Flink and am encountering a nasty problem while trying to submit a streaming Flink Job. I'll try to explain it as thoroughly as possible.

Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as per documentation and I submit the job using yarn-session.sh and then flink run.

Getting into more details, the Flink Job is a fat jar built with sbt assembly and the following dependencies:

libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion % "provided"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"


assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

inside the flink lib folder I have the following Jars:

flink-connector-filesystem_2.10-1.3.2.jar 

flink-python_2.10-1.3.2.jar
flink-connector-kinesis_2.10-1.3.2.jar    

flink-shaded-hadoop2-uber-1.3.2.jar
flink-dist_2.10-1.3.2.jar                 

log4j-1.2.17.jar
flink-hbase_2.10-1.3.2.jar               

slf4j-log4j12-1.7.7.jar

Right now, the issue is that when I flink run the job, I get NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it being inside the fat jar. So, I tried putting both hbase-common.jar and hbase-client.jars inside the lib folder, getting as a result another NoClassDefFound, only, this time, for com/google/common/collect/ListMultimap. Now, I noticed that flink-connector-kinesis and all the hbase packages (both hbase-common and flink-hbase deps) have a different version of guava as dependency: the first one needs version 18.0, while the second one needs 12.0.1 version. I, then, tried to enable the shading of guava packages with the following:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
)

Only to no avail, getting the exact same error as before. And I really don't know how to submit the job without pouring every single dependency inside the lib folder. I thought it could be a compatibility issue with the YARN cluster, but I tried launching the job locally only to get the same error.

As an additional info, I get the following warnings if I don't specify flink packages as provided:

[warn]     * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over 2.0.1
[warn]         +- org.typelevel:macro-compat_2.10:1.1.1              (depends on 2.1.0)
[warn]         +- org.apache.flink:flink-scala_2.10:1.3.2            (depends on 2.0.1)
[warn]
[warn]     * org.mortbay.jetty:jetty-util:6.1.26 is selected over 6.1.26.hwx
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 6.1.26)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2        (depends on 6.1.26.hwx)
[warn]
[warn]     * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
[warn]         +- com.data-artisans:flakka-remote_2.10:2.3-custom    (depends on 3.8.0.Final)
[warn]         +- org.apache.zookeeper:zookeeper:3.4.6               (depends on 3.7.0.Final)
[warn]
[warn]     * com.google.guava:guava:18.0 is selected over 12.0.1
[warn]         +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2 (depends on 18.0)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0             (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 12.0.1)
[warn]
[warn]     * io.netty:netty-all:4.0.27.Final is selected over 4.0.23.Final
[warn]         +- org.apache.flink:flink-runtime_2.10:1.3.2          (depends on 4.0.27.Final)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2        (depends on 4.0.27.Final)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 4.0.23.Final)
[warn]
[warn]     * junit:junit:4.12 is selected over 3.8.1
[warn]         +- org.apache.hbase:hbase-protocol:1.3.0              (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-annotations:1.3.0           (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0           (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0             (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-client:1.3.0                (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-common:1.3.0                (depends on 4.12)
[warn]         +- org.apache.hbase:hbase-server:1.3.0                (depends on 4.12)
[warn]         +- jline:jline:0.9.94                                 (depends on 3.8.1)

And here I am asking for help here.

Thank you very much for your attention,

Kind regards,

Federico D'Ambrosio