Flink 0.10.2 and Kafka 0.8.1

[Robert Schmidtke]

Hi everyone,

I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error.

I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly?

Thanks!

Robert

--

[Balaji Rajagopalan]

I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. 

balaji 

On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <[hidden email]> wrote:

Hi everyone,

I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error.

I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly?

Thanks!

Robert

--

My GPG Key ID: 336E2680

[rmetzger0]

Hi,

did you check your user jar if it contains the Kafka classes?

Are you building a fat jar? Are you manually excluding any dependencies?

Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" dependency also contains the org.apache.kafka.common.Node class (the LegacyFetcher needs that class).

[1] https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom

[2] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom

On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <[hidden email]> wrote:

I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. 

balaji 

On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <[hidden email]> wrote:

Hi everyone,

I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error.

I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly?

Thanks!

Robert

--

My GPG Key ID: 336E2680

... [show rest of quote]

[Robert Schmidtke]

Hi Robert,

thanks for your hints. I was not sure whether I was building a proper fat jar, as I have not used the Flink Archetype for my project. However, I have set up a sample project at

https://github.com/robert-schmidtke/flink-test/

which is nothing more than the Quickstart Archetype plus the instructions to bundle the Kafka dependencies. The resulting jars (mvn clean package -Pbuild-jar and mvn clean package) do not contain the org/apache/kafka/** classes. Can you have a quick look at the pom? However, as I said, it's verbatim Archetype+Flink Docs.

Thanks a lot in advance!

Robert

On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <[hidden email]> wrote:

Hi,

did you check your user jar if it contains the Kafka classes?

Are you building a fat jar? Are you manually excluding any dependencies?

Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" dependency also contains the org.apache.kafka.common.Node class (the LegacyFetcher needs that class).

[1] https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom

[2] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom

On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <[hidden email]> wrote:

I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. 

balaji 

On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <[hidden email]> wrote:

Hi everyone,

I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error.

I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly?

Thanks!

Robert

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

[rmetzger0]

Hi,

the problem with the posted project is that it doesn't have the Flink kafka connector as a dependency.

On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke <[hidden email]> wrote:

Hi Robert,

thanks for your hints. I was not sure whether I was building a proper fat jar, as I have not used the Flink Archetype for my project. However, I have set up a sample project at

https://github.com/robert-schmidtke/flink-test/

which is nothing more than the Quickstart Archetype plus the instructions to bundle the Kafka dependencies. The resulting jars (mvn clean package -Pbuild-jar and mvn clean package) do not contain the org/apache/kafka/** classes. Can you have a quick look at the pom? However, as I said, it's verbatim Archetype+Flink Docs.

Thanks a lot in advance!

Robert

On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <[hidden email]> wrote:

Hi,

did you check your user jar if it contains the Kafka classes?

Are you building a fat jar? Are you manually excluding any dependencies?

Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" dependency also contains the org.apache.kafka.common.Node class (the LegacyFetcher needs that class).

[1] https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom

[2] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom

On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <[hidden email]> wrote:

I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. 

balaji 

On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <[hidden email]> wrote:

Hi everyone,

I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error.

I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly?

Thanks!

Robert

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

[Robert Schmidtke]

You're right, it does not. When including it the resulting jar has the Kafka dependencies bundled. Now it's up to me to figure out the difference between the sample project and the one I'm working on.

Thanks! Really quick help.

Robert

On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger <[hidden email]> wrote:

Hi,

the problem with the posted project is that it doesn't have the Flink kafka connector as a dependency.

On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke <[hidden email]> wrote:

Hi Robert,

thanks for your hints. I was not sure whether I was building a proper fat jar, as I have not used the Flink Archetype for my project. However, I have set up a sample project at

https://github.com/robert-schmidtke/flink-test/

which is nothing more than the Quickstart Archetype plus the instructions to bundle the Kafka dependencies. The resulting jars (mvn clean package -Pbuild-jar and mvn clean package) do not contain the org/apache/kafka/** classes. Can you have a quick look at the pom? However, as I said, it's verbatim Archetype+Flink Docs.

Thanks a lot in advance!

Robert

On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <[hidden email]> wrote:

Hi,

did you check your user jar if it contains the Kafka classes?

Are you building a fat jar? Are you manually excluding any dependencies?

Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" dependency also contains the org.apache.kafka.common.Node class (the LegacyFetcher needs that class).

[1] https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom

[2] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom

On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <[hidden email]> wrote:

I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. 

balaji 

On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <[hidden email]> wrote:

Hi everyone,

I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error.

I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly?

Thanks!

Robert

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

[rmetzger0]

If you want you can share the pom of your project privately.

On Mon, Apr 18, 2016 at 4:05 PM, Robert Schmidtke <[hidden email]> wrote:

You're right, it does not. When including it the resulting jar has the Kafka dependencies bundled. Now it's up to me to figure out the difference between the sample project and the one I'm working on.

Thanks! Really quick help.

Robert

On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger <[hidden email]> wrote:

Hi,

the problem with the posted project is that it doesn't have the Flink kafka connector as a dependency.

On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke <[hidden email]> wrote:

Hi Robert,

thanks for your hints. I was not sure whether I was building a proper fat jar, as I have not used the Flink Archetype for my project. However, I have set up a sample project at

https://github.com/robert-schmidtke/flink-test/

which is nothing more than the Quickstart Archetype plus the instructions to bundle the Kafka dependencies. The resulting jars (mvn clean package -Pbuild-jar and mvn clean package) do not contain the org/apache/kafka/** classes. Can you have a quick look at the pom? However, as I said, it's verbatim Archetype+Flink Docs.

Thanks a lot in advance!

Robert

On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <[hidden email]> wrote:

Hi,

did you check your user jar if it contains the Kafka classes?

Are you building a fat jar? Are you manually excluding any dependencies?

Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" dependency also contains the org.apache.kafka.common.Node class (the LegacyFetcher needs that class).

[1] https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom

[2] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom

On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <[hidden email]> wrote:

I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. 

balaji 

On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <[hidden email]> wrote:

Hi everyone,

I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error.

I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly?

Thanks!

Robert

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

[Robert Schmidtke]

Hi Robert,

thanks for your offer. After playing around a bit I would like to take it, if you have the time: https://github.com/robert-schmidtke/HiBench/blob/flink-streaming/src/streambench/flinkbench/pom.xml

I would guess the POM is similar to the one in the sample project, yet when building it, the jar does not contain all indirect dependencies.

Thanks!!

Robert

On Mon, Apr 18, 2016 at 4:07 PM, Robert Metzger <[hidden email]> wrote:

If you want you can share the pom of your project privately.

On Mon, Apr 18, 2016 at 4:05 PM, Robert Schmidtke <[hidden email]> wrote:

You're right, it does not. When including it the resulting jar has the Kafka dependencies bundled. Now it's up to me to figure out the difference between the sample project and the one I'm working on.

Thanks! Really quick help.

Robert

On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger <[hidden email]> wrote:

Hi,

the problem with the posted project is that it doesn't have the Flink kafka connector as a dependency.

On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke <[hidden email]> wrote:

Hi Robert,

thanks for your hints. I was not sure whether I was building a proper fat jar, as I have not used the Flink Archetype for my project. However, I have set up a sample project at

https://github.com/robert-schmidtke/flink-test/

which is nothing more than the Quickstart Archetype plus the instructions to bundle the Kafka dependencies. The resulting jars (mvn clean package -Pbuild-jar and mvn clean package) do not contain the org/apache/kafka/** classes. Can you have a quick look at the pom? However, as I said, it's verbatim Archetype+Flink Docs.

Thanks a lot in advance!

Robert

On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <[hidden email]> wrote:

Hi,

did you check your user jar if it contains the Kafka classes?

Are you building a fat jar? Are you manually excluding any dependencies?

Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" dependency also contains the org.apache.kafka.common.Node class (the LegacyFetcher needs that class).

[1] https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom

[2] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom

On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <[hidden email]> wrote:

I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. 

balaji 

On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <[hidden email]> wrote:

Hi everyone,

I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error.

I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly?

Thanks!

Robert

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

[Robert Schmidtke]

Turns out when I remove the explicit dependency on kafka_2.10 v. 0.8.1, then the dependencies are properly included. Guess there was a conflict somehow? I'll need to figure out if the rest of the code is fine with kafka_2.10 v. 0.8.2.0 as well.

On Mon, Apr 18, 2016 at 4:32 PM, Robert Schmidtke <[hidden email]> wrote:

Hi Robert,

thanks for your offer. After playing around a bit I would like to take it, if you have the time: https://github.com/robert-schmidtke/HiBench/blob/flink-streaming/src/streambench/flinkbench/pom.xml

I would guess the POM is similar to the one in the sample project, yet when building it, the jar does not contain all indirect dependencies.

Thanks!!

Robert

On Mon, Apr 18, 2016 at 4:07 PM, Robert Metzger <[hidden email]> wrote:

If you want you can share the pom of your project privately.

On Mon, Apr 18, 2016 at 4:05 PM, Robert Schmidtke <[hidden email]> wrote:

You're right, it does not. When including it the resulting jar has the Kafka dependencies bundled. Now it's up to me to figure out the difference between the sample project and the one I'm working on.

Thanks! Really quick help.

Robert

On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger <[hidden email]> wrote:

Hi,

the problem with the posted project is that it doesn't have the Flink kafka connector as a dependency.

On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke <[hidden email]> wrote:

Hi Robert,

thanks for your hints. I was not sure whether I was building a proper fat jar, as I have not used the Flink Archetype for my project. However, I have set up a sample project at

https://github.com/robert-schmidtke/flink-test/

which is nothing more than the Quickstart Archetype plus the instructions to bundle the Kafka dependencies. The resulting jars (mvn clean package -Pbuild-jar and mvn clean package) do not contain the org/apache/kafka/** classes. Can you have a quick look at the pom? However, as I said, it's verbatim Archetype+Flink Docs.

Thanks a lot in advance!

Robert

On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <[hidden email]> wrote:

Hi,

did you check your user jar if it contains the Kafka classes?

Are you building a fat jar? Are you manually excluding any dependencies?

Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" dependency also contains the org.apache.kafka.common.Node class (the LegacyFetcher needs that class).

[1] https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom

[2] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom

On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <[hidden email]> wrote:

I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. 

balaji 

On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <[hidden email]> wrote:

Hi everyone,

I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error.

I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly?

Thanks!

Robert

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

--

[rmetzger0]

Yeah, that's the problem

Your Kafka 0.8.1 dependency [1] doesn't depend on kafka-clients. With the explicit dependency definition, you are overriding the transitive Kafka dependency from the kafka connector.

[1] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.1/kafka_2.10-0.8.1.pom

On Mon, Apr 18, 2016 at 5:38 PM, Robert Schmidtke <[hidden email]> wrote:

Turns out when I remove the explicit dependency on kafka_2.10 v. 0.8.1, then the dependencies are properly included. Guess there was a conflict somehow? I'll need to figure out if the rest of the code is fine with kafka_2.10 v. 0.8.2.0 as well.

On Mon, Apr 18, 2016 at 4:32 PM, Robert Schmidtke <[hidden email]> wrote:

Hi Robert,

thanks for your offer. After playing around a bit I would like to take it, if you have the time: https://github.com/robert-schmidtke/HiBench/blob/flink-streaming/src/streambench/flinkbench/pom.xml

I would guess the POM is similar to the one in the sample project, yet when building it, the jar does not contain all indirect dependencies.

Thanks!!

Robert

On Mon, Apr 18, 2016 at 4:07 PM, Robert Metzger <[hidden email]> wrote:

If you want you can share the pom of your project privately.

On Mon, Apr 18, 2016 at 4:05 PM, Robert Schmidtke <[hidden email]> wrote:

You're right, it does not. When including it the resulting jar has the Kafka dependencies bundled. Now it's up to me to figure out the difference between the sample project and the one I'm working on.

Thanks! Really quick help.

Robert

On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger <[hidden email]> wrote:

Hi,

the problem with the posted project is that it doesn't have the Flink kafka connector as a dependency.

On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke <[hidden email]> wrote:

Hi Robert,

thanks for your hints. I was not sure whether I was building a proper fat jar, as I have not used the Flink Archetype for my project. However, I have set up a sample project at

https://github.com/robert-schmidtke/flink-test/

which is nothing more than the Quickstart Archetype plus the instructions to bundle the Kafka dependencies. The resulting jars (mvn clean package -Pbuild-jar and mvn clean package) do not contain the org/apache/kafka/** classes. Can you have a quick look at the pom? However, as I said, it's verbatim Archetype+Flink Docs.

Thanks a lot in advance!

Robert

On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <[hidden email]> wrote:

Hi,

did you check your user jar if it contains the Kafka classes?

Are you building a fat jar? Are you manually excluding any dependencies?

Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" dependency also contains the org.apache.kafka.common.Node class (the LegacyFetcher needs that class).

[1] https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom

[2] https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom

On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <[hidden email]> wrote:

I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. 

balaji 

On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <[hidden email]> wrote:

Hi everyone,

I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error.

I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly?

Thanks!

Robert

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]

--

My GPG Key ID: 336E2680

... [show rest of quote]