Flink 0.10.2 and Kafka 0.8.1
Locked
Flink 0.10.2 and Kafka 0.8.1
 
|
Hi everyone,
I have a Kafka cluster running on version 0.8.1, hence I'm using the FlinkKafkaConsumer081. When running my program, I saw a NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my binaries according to https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution, however I'm still seeing the error. I played around a bit and it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an error in the documentation or have I not figured out something properly? Thanks! Robert My GPG Key ID: 336E2680 |
Re: Flink 0.10.2 and Kafka 0.8.1
|
|
I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was never able to get it working confounded with noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2 scala version 2.11 things worked for me, if moving to flink 1.0.0 is an option for you do so. balaji On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <[hidden email]> wrote:
|
|
|
Hi, did you check your user jar if it contains the Kafka classes? Are you building a fat jar? Are you manually excluding any dependencies? On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <[hidden email]> wrote:
|
Re: Flink 0.10.2 and Kafka 0.8.1
|
|
Hi Robert, thanks for your hints. I was not sure whether I was building a proper fat jar, as I have not used the Flink Archetype for my project. However, I have set up a sample project at which is nothing more than the Quickstart Archetype plus the instructions to bundle the Kafka dependencies. The resulting jars (mvn clean package -Pbuild-jar and mvn clean package) do not contain the org/apache/kafka/** classes. Can you have a quick look at the pom? However, as I said, it's verbatim Archetype+Flink Docs. Thanks a lot in advance! Robert On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <[hidden email]> wrote:
My GPG Key ID: 336E2680 |
|
|
Hi, the problem with the posted project is that it doesn't have the Flink kafka connector as a dependency. On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke <[hidden email]> wrote:
|
Re: Flink 0.10.2 and Kafka 0.8.1
|
|
You're right, it does not. When including it the resulting jar has the Kafka dependencies bundled. Now it's up to me to figure out the difference between the sample project and the one I'm working on. Thanks! Really quick help. Robert On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger <[hidden email]> wrote:
My GPG Key ID: 336E2680 |
|
|
If you want you can share the pom of your project privately. On Mon, Apr 18, 2016 at 4:05 PM, Robert Schmidtke <[hidden email]> wrote:
|
Re: Flink 0.10.2 and Kafka 0.8.1
|
|
Hi Robert, thanks for your offer. After playing around a bit I would like to take it, if you have the time: https://github.com/robert-schmidtke/HiBench/blob/flink-streaming/src/streambench/flinkbench/pom.xml I would guess the POM is similar to the one in the sample project, yet when building it, the jar does not contain all indirect dependencies. Thanks!! Robert On Mon, Apr 18, 2016 at 4:07 PM, Robert Metzger <[hidden email]> wrote:
My GPG Key ID: 336E2680 |
Re: Flink 0.10.2 and Kafka 0.8.1
|
|
Turns out when I remove the explicit dependency on kafka_2.10 v. 0.8.1, then the dependencies are properly included. Guess there was a conflict somehow? I'll need to figure out if the rest of the code is fine with kafka_2.10 v. 0.8.2.0 as well. On Mon, Apr 18, 2016 at 4:32 PM, Robert Schmidtke <[hidden email]> wrote:
My GPG Key ID: 336E2680 |
|
|
Yeah, that's the problem Your Kafka 0.8.1 dependency [1] doesn't depend on kafka-clients. With the explicit dependency definition, you are overriding the transitive Kafka dependency from the kafka connector. On Mon, Apr 18, 2016 at 5:38 PM, Robert Schmidtke <[hidden email]> wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |