Flink Kafka runtime error

[Wendong]

Hello,

I am using Flink 0.9, Scala 2.10.4, Kafka 0.8.2.1 and trying to consume Kafka messages in Flink.

Here is the build.sbt:
 
scalaVersion := "2.10.4"

libraryDependencies += "org.apache.flink" % "flink-connector-kafka" % "0.9.0" exclude("org.apache.kafka", "kafka_${scala.binary.version}")
libraryDependencies += "org.apache.kafka" %% "kafka" % "0.8.2.1"

My program TestKafka.scala is simple:

import org.apache.flink.streaming.api.environment._
import org.apache.flink.streaming.connectors.kafka
import org.apache.flink.streaming.connectors.kafka.api._
import org.apache.flink.streaming.util.serialization._

object TestKafka {
  def main(args: Array[String]) {
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    val stream = env
     .addSource(new KafkaSource[String]("localhost:2181", "test", new SimpleStringSchema))
     .print
  }
}

Compilation is fine. However, when I run the jar file using:
./bin/flink run /path/to/project/target/scala-2.10/TestKafka_2.10-1.0.jar, I got runtime errors:

java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/api/KafkaSource
        at TestKafka$.main(TestKafka.scala:10)
        at TestKafka.main(TestKafka.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
.................
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.api.KafkaSource
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
.........

Any additional setting is missing?

Thanks for help!

Wendong

[Wendong]

Just found a workaround. I downloaded kafka_2.10-0.8.2.1.jar and flink-connector-kafka-0.9.0.jar, then put them into $FLINK_HOME/lib/. Now the runtime error is gone. But this is just a workaound. I believe there is a better solution.

Wendong

[Stephan Ewen]

Hi Wendong!

The streaming connectors are not in Flink's "system classpath", because they depend on many libraries (zookeeper, asm, protocol buffers), and we want to keep the default dependencies slim. This reduces version conflicts for people where the user code depends on these libraries.

As a consequence, you need to do one of the two things when using a connector:

  - The recommended solution is to build a fat jar of you user code and the connector (with its dependencies). The maven quickstart archetypes take care of that.

  - The other solution (if you do not want to build a fat jar) is to manually add the connector code to the Flink lib directory (as you did).

Greetings,

Stephan

On Wed, Jul 15, 2015 at 9:01 PM, Wendong <[hidden email]> wrote:

Just found a workaround. I downloaded kafka_2.10-0.8.2.1.jar and
flink-connector-kafka-0.9.0.jar, then put them into $FLINK_HOME/lib/. Now
the runtime error is gone. But this is just a workaound. I believe there is
a better solution.

Wendong



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-runtime-error-tp2063p2064.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Wendong]

Hi Stephan,

Thanks for your suggestion! I'll try the first solution too.

Wendong