jackson DefaultScalaModule missing on deployment

[Bart van Deenen]

Hi all

I've succesfully built a Flink streaming job, and it runs beautifully in
my IntelliJ ide, with a Flink instance started on the fly. The job eats
Kafka events, and outputs to file. All the i/o is json encoded with
Jackson.

But I'm having trouble with deploying the jar on a Flink server
(version 1.0.0)
Building the jar via sbt package generates a jar with only my classes in
it, and the Flink cluster misses some Jackson stuff, and it complains
when I try to run the jar file.

java.lang.NoClassDefFoundError:
com/fasterxml/jackson/module/scala/DefaultScalaModule

The build.sbt is pretty much identical to the quick-start:

resolvers in ThisBuild ++= Seq(Resolver.mavenLocal)
name := "Flink Project"
version := "0.1-SNAPSHOT"
organization := "org.example"
scalaVersion in ThisBuild := "2.11.7"
val flinkVersion = "1.0.0"

val flinkDependencies = Seq(
  "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
  "org.apache.flink" %% "flink-streaming-scala" % flinkVersion %
  "provided")
lazy val root = (project in file(".")).
  settings(
    libraryDependencies ++= flinkDependencies
  )
libraryDependencies += "org.apache.flink" %
"flink-connector-kafka-0.9_2.11" % "1.0.0"
libraryDependencies += "com.fasterxml.jackson.module" %
"jackson-module-scala_2.11" % "2.4.5"
mainClass in assembly := Some("com.kpn.cj.journeyengine.Streaming")

// make run command include the provided dependencies
run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass
in (Compile, run), runner in (Compile, run))

// exclude Scala library from assembly
assemblyOption in assembly := (assemblyOption in
assembly).value.copy(includeScala = false)

I'm not a hero as far as Scala Build Tool is concerned, any hints on how
to make sure the jackson stuff is actually included in the flink job?

Thanks

Bart

[Balaji Rajagopalan]

You will have to include dependent jackson jar in flink server lib folder, or create a fat jar. 

balaji 

On Tue, Mar 29, 2016 at 4:47 PM, Bart van Deenen <[hidden email]> wrote:

Hi all

I've succesfully built a Flink streaming job, and it runs beautifully in
my IntelliJ ide, with a Flink instance started on the fly. The job eats
Kafka events, and outputs to file. All the i/o is json encoded with
Jackson.

But I'm having trouble with deploying the jar on a Flink server
(version 1.0.0)
Building the jar via sbt package generates a jar with only my classes in
it, and the Flink cluster misses some Jackson stuff, and it complains
when I try to run the jar file.

java.lang.NoClassDefFoundError:
com/fasterxml/jackson/module/scala/DefaultScalaModule

The build.sbt is pretty much identical to the quick-start:

resolvers in ThisBuild ++= Seq(Resolver.mavenLocal)
name := "Flink Project"
version := "0.1-SNAPSHOT"
organization := "org.example"
scalaVersion in ThisBuild := "2.11.7"
val flinkVersion = "1.0.0"

val flinkDependencies = Seq(
  "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
  "org.apache.flink" %% "flink-streaming-scala" % flinkVersion %
  "provided")
lazy val root = (project in file(".")).
  settings(
    libraryDependencies ++= flinkDependencies
  )
libraryDependencies += "org.apache.flink" %
"flink-connector-kafka-0.9_2.11" % "1.0.0"
libraryDependencies += "com.fasterxml.jackson.module" %
"jackson-module-scala_2.11" % "2.4.5"
mainClass in assembly := Some("com.kpn.cj.journeyengine.Streaming")

// make run command include the provided dependencies
run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass
in (Compile, run), runner in (Compile, run))

// exclude Scala library from assembly
assemblyOption in assembly := (assemblyOption in
assembly).value.copy(includeScala = false)

I'm not a hero as far as Scala Build Tool is concerned, any hints on how
to make sure the jackson stuff is actually included in the flink job?

Thanks

Bart

[Bart van Deenen]

Thanks

Copying jars into the lib directory works fine

 

 

On Tue, Mar 29, 2016, at 13:34, Balaji Rajagopalan wrote:

You will have to include dependent jackson jar in flink server lib folder, or create a fat jar. 

 

balaji 

 

On Tue, Mar 29, 2016 at 4:47 PM, Bart van Deenen <[hidden email]> wrote:

Hi all

 

I've succesfully built a Flink streaming job, and it runs beautifully in

my IntelliJ ide, with a Flink instance started on the fly. The job eats

Kafka events, and outputs to file. All the i/o is json encoded with

Jackson.

 

But I'm having trouble with deploying the jar on a Flink server

(version 1.0.0)

Building the jar via sbt package generates a jar with only my classes in

it, and the Flink cluster misses some Jackson stuff, and it complains

when I try to run the jar file.

 

java.lang.NoClassDefFoundError:

com/fasterxml/jackson/module/scala/DefaultScalaModule

 

The build.sbt is pretty much identical to the quick-start:

 

resolvers in ThisBuild ++= Seq(Resolver.mavenLocal)

name := "Flink Project"

version := "0.1-SNAPSHOT"

organization := "org.example"

scalaVersion in ThisBuild := "2.11.7"

val flinkVersion = "1.0.0"

 

val flinkDependencies = Seq(

"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",

"org.apache.flink" %% "flink-streaming-scala" % flinkVersion %

"provided")

lazy val root = (project in file(".")).

settings(

libraryDependencies ++= flinkDependencies

)

libraryDependencies += "org.apache.flink" %

"flink-connector-kafka-0.9_2.11" % "1.0.0"

libraryDependencies += "com.fasterxml.jackson.module" %

"jackson-module-scala_2.11" % "2.4.5"

mainClass in assembly := Some("com.kpn.cj.journeyengine.Streaming")

 

// make run command include the provided dependencies

run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass

in (Compile, run), runner in (Compile, run))

 

// exclude Scala library from assembly

assemblyOption in assembly := (assemblyOption in

assembly).value.copy(includeScala = false)

 

I'm not a hero as far as Scala Build Tool is concerned, any hints on how

to make sure the jackson stuff is actually included in the flink job?

 

Thanks

Bart