(DEPRECATED) Apache Flink User Mailing List archive.

Re: Starting Flink cluster and running a job

Posted by Boris Lublinsky on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Jira-issue-Flink-11127-tp26180p26294.html

I am doing this using sbt native packager and sbt docker.

ef sbtdockerFlinkAppBase(id: String)(base: String = id) = projectBase(id)(base)
  .enablePlugins(sbtdocker.DockerPlugin)
  .settings(
    dockerfile in docker := {

      val artifact: File = assembly.value
      // This location is fixed. the jar has to go there so that Flink can load it
      val artifactTargetPath = s"/opt/flink/lib/${artifact.name}"

      new Dockerfile {
        from ("lightbend/flink:1.7.1-scala_2.11")
        add(artifact, artifactTargetPath)
      }
    },

    // Set name for the image
    imageNames in docker := Seq(
      ImageName(namespace = Some(organization.value),
        repository = name.value.toLowerCase, 
        tag = Some(version.value))
    ),

    buildOptions in docker := BuildOptions(cache = false)
  )

It produce docker file as follows

FROM lightbend/flink:1.7.1-scala_2.11
ADD 0/fdp-flink-taxiride-assembly-2.0.0.jar /opt/flink/lib/fdp-flink-taxiride-assembly-2.0.0.jar

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <[hidden email]> wrote:
Hi,
One additional question, how do you actually build the docker image? How do you put the user jar into the container? Maybe you added extra kafka connector to the cluster classpath? Have you checked what is on the classpath of a running taskmanager?

Best,
Dawid

On 22/02/2019 15:44, Boris Lublinsky wrote:
Gordon, I double checked it several times. Here is the list

Columns are

[info] - Jar-Size including dependencies

[info] - Jar-Size

[info] - Number of transitive dependencies

[info] - Number of direct dependencies

[info] - ModuleID

[info] Done updating.

[info] TotSize JarSize #TDe #Dep Module

[info] 27.988 MB ------- MB 33 2 lightbend:fdp-flink-taxiride_2.11:2.0.0

[info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0

[info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0

[info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0

[info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0

[info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0

[info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0

[info] 8.397 MB 0.068 MB 8 5 org.apache.flink:flink-connector-kafka_2.11:1.7.1

[info] 8.175 MB 1.889 MB 4 4 org.apache.kafka:kafka-clients:2.1.0

[info] 7.973 MB 0.212 MB 7 4 org.typelevel:cats-kernel-laws_2.11:0.9.0

[info] 4.834 MB 0.007 MB 4 2 org.typelevel:cats-macros_2.11:0.9.0

[info] 4.746 MB 0.082 MB 2 2 com.typesafe.scala-logging:scala-logging_2.11:3.9.0

[info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1

[info] 4.623 MB 4.623 MB 0 0 org.scala-lang:scala-reflect:2.11.12

[info] 4.592 MB 4.592 MB 0 0 org.typelevel:cats-kernel_2.11:0.9.0

[info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4

[info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2

[info] 3.109 MB 3.094 MB 1 1 org.scalacheck:scalacheck_2.11:1.13.4

[info] 2.019 MB 2.019 MB 0 0 org.xerial.snappy:snappy-java:1.1.7.2

[info] 0.803 MB 0.290 MB 2 2 ch.qos.logback:logback-classic:1.2.3

[info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1

[info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0

[info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3

[info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3

[info] 0.195 MB 0.114 MB 3 3 org.apache.flink:flink-connector-kafka-base_2.11:1.7.1

[info] 0.170 MB 0.167 MB 1 1 com.github.mpilquist:simulacrum_2.11:0.10.0

[info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2

[info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25

[info] 0.033 MB 0.033 MB 0 0 com.google.code.findbugs:jsr305:1.3.9

[info] 0.016 MB 0.002 MB 2 1 org.typelevel:catalysts-platform_2.11:0.0.5

[info] 0.015 MB 0.012 MB 1 1 org.typelevel:catalysts-macros_2.11:0.0.5

[info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0

[info] 0.007 MB 0.007 MB 0 0 org.apache.flink:force-shading:1.7.1

[info] 0.003 MB 0.003 MB 0 0 org.typelevel:macro-compat_2.11:1.1.1

And here is another view

[info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]

[info] +-lightbend:support_2.11:2.0.0 [S]

[info] | +-ch.qos.logback:logback-classic:1.2.3

[info] | | +-ch.qos.logback:logback-core:1.2.3

[info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)

[info] | | +-org.slf4j:slf4j-api:1.7.25

[info] | |

[info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)

[info] | | +-org.slf4j:slf4j-api:1.7.25

[info] | |

[info] | +-com.typesafe:config:1.3.3

[info] | +-joda-time:joda-time:2.10.1

[info] | +-org.joda:joda-convert:2.1.2

[info] | +-org.typelevel:cats_2.11:0.9.0 [S]

[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | |

[info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]

[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | | |

[info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | | |

[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | |

[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]

[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | | |

[info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | | |

[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | |

[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]

[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | | +-org.scala-sbt:test-interface:1.0

[info] | | |

[info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]

[info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]

[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | +-org.scala-sbt:test-interface:1.0

[info] | |

[info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | | +-org.scala-sbt:test-interface:1.0

[info] | | |

[info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]

[info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]

[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | | |

[info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | | |

[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | |

[info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]

[info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | | | +-org.scala-sbt:test-interface:1.0

[info] | | | |

[info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]

[info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]

[info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | | +-org.scala-sbt:test-interface:1.0

[info] | | |

[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | |

[info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]

[info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | | +-org.scala-sbt:test-interface:1.0

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] |

[info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1

[info] +-com.google.code.findbugs:jsr305:1.3.9

[info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1

[info] | +-com.google.code.findbugs:jsr305:1.3.9

[info] | +-org.apache.flink:force-shading:1.7.1

[info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)

[info] | +-org.slf4j:slf4j-api:1.7.25

[info] |

[info] +-org.apache.flink:force-shading:1.7.1

[info] +-org.apache.kafka:kafka-clients:2.1.0

[info] | +-com.github.luben:zstd-jni:1.3.5-4

[info] | +-org.lz4:lz4-java:1.5.0

[info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)

[info] | +-org.slf4j:slf4j-api:1.7.25

[info] | +-org.xerial.snappy:snappy-java:1.1.7.2

[info] |

[info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)

[info] +-org.slf4j:slf4j-api:1.7.25

[info]

[info] Done updating.

[info] lightbend:support_2.11:2.0.0 [S]

[info] +-ch.qos.logback:logback-classic:1.2.3

[info] | +-ch.qos.logback:logback-core:1.2.3

[info] | +-org.slf4j:slf4j-api:1.7.25

[info] |

[info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]

[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | +-org.slf4j:slf4j-api:1.7.25

[info] |

[info] +-com.typesafe:config:1.3.3

[info] +-joda-time:joda-time:2.10.1

[info] +-org.joda:joda-convert:2.1.2

[info] +-org.typelevel:cats_2.11:0.9.0 [S]

[info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] |

[info] +-org.typelevel:cats-core_2.11:0.9.0 [S]

[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | |

[info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] |

[info] +-org.typelevel:cats-free_2.11:0.9.0 [S]

[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | |

[info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] |

[info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]

[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | |

[info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] |

[info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]

[info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | +-org.scala-sbt:test-interface:1.0

[info] | |

[info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]

[info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]

[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | |

[info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | +-org.typelevel:discipline_2.11:0.7.2 [S]

[info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | +-org.scala-sbt:test-interface:1.0

[info] |

[info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]

[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | |

[info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | +-org.scala-sbt:test-interface:1.0

[info] | |

[info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]

[info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]

[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | |

[info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | | |

[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]

[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | | +-org.scala-sbt:test-interface:1.0

[info] | | |

[info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]

[info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]

[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | +-org.scala-sbt:test-interface:1.0

[info] | |

[info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]

[info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | | |

[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] | |

[info] | +-org.typelevel:discipline_2.11:0.7.2 [S]

[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]

[info] | | +-org.scala-sbt:test-interface:1.0

[info] | |

[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] |

[info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]

[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]

[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]

[info] | |

[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]

[info] |

[info] +-org.typelevel:machinist_2.11:0.6.1 [S]

[info] +-org.scala-lang:scala-reflect:2.11.12 [S]

[info]
Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/
On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi,

I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a reoccurrence of FLINK-8741.
I've double checked the used classloaders in the Kafka connector, they seem to be correct.

The fact that it works correctly in IntelliJ, but not when packaged, suggests that there could be some conflicting dependencies in the packaged jar.

Could you check the actual resolved dependency tree of the project, and see if there are multiple versions of some dependency related to Kafka being pulled in? For Maven for example, that would be "mvn dependency:tree".

Cheers,
Gordon
On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <[hidden email]> wrote:
The relevant dependencies are
val flinkScala            =      "org.apache.flink"             %%   "flink-scala"                    % flinkVersion % "provided"
val flinkStreamingScala   =      "org.apache.flink"             %%   "flink-streaming-scala"          % flinkVersion % "provided"
val flinkKafka            =      "org.apache.flink"             %%   "flink-connector-kafka"          % flinkVersion exclude("org.slf4j", "slf4j-log4j12") 
I am using SBT
I tried both connector-kafka and connector-kaka-11 - same result

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/
On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <[hidden email]> wrote:
Hi Boris,

can you the relevant parts (dependencies) of your pom.xml? Did you also try without fixing the Kafka version, i.e. running with the Kafka client version provided by the Kafka connector of Flink? Gordon (cc) dealt with FLINK-8741.

@Gordon: have you seen this issue with 1.6/1.7 before?

Cheers,

Konstantin
On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <[hidden email]> wrote:
I found some more details on this

The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@...%3E

And was due to https://issues.apache.org/jira/browse/FLINK-8741

It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector

And Kafka-connector-011

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/
On Feb 19, 2019, at 7:02 PM, Ken Krugler <[hidden email]> wrote:
Hi Boris,

I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.

When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.

Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.

— Ken

PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html
On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <[hidden email]> wrote:
Konstantin,
After experimenting with this for a while, I got to the root cause of the problem

I am running a version of a Taxi ride travel prediction as my sample.

It works fine in Intellij,

But when I am trying to put it in the docker (standard Debian 1.7 image)

It fails with a following error

The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)

at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)

at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)

at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)

at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)

at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)

at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)

at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)

at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)

at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)

at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)

at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)

at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)

Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.

at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)

at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)

... 19 more

Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer

at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)

at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)

at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)

at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)

at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)

at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)

at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)

at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)

at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)

at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)

at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)

at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)

at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)

at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)

at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer

at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)

at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)

... 17 more

The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no

Which talks about class loader. (I tried there solution, but it did not help)

I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.

Have you guys seen this error before?

Any suggestion?
Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/
On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <[hidden email]> wrote:
Hi Boris,

without looking at the entrypoint in much detail, generally there should not be a race condition there:

* if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)

* if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.

So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.

Could you provide the Taskmanager/JobManager logs for such a failure case?

Cheers,

Konstantin
On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <[hidden email]> wrote:
Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker
I have created an entry point, which looks like follows:
#!/bin/sh

################################################################################
#   from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh
#   and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh
################################################################################

# If unspecified, the hostname of the container is taken as the JobManager address
JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}

drop_privs_cmd() {
    if [ $(id -u) != 0 ]; then
        # Don't need to drop privs if EUID != 0
        return
    elif [ -x /sbin/su-exec ]; then
        # Alpine
        echo su-exec flink
    else
        # Others
        echo gosu flink
    fi
}

JOB_MANAGER="jobmanager"
TASK_MANAGER="taskmanager"

CMD="$1"
shift

if [ "${CMD}" = "help" ]; then
    echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
    exit 0
elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
    if [ "${CMD}" = "${TASK_MANAGER}" ]; then
        TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}

        sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
        sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
        echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
        echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"

        echo "Starting Task Manager"
        echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
        exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
    else
        sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
        echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
        echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
        echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"

        if [ -z "$1" ]; then
           exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
        else
            exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
        fi
    fi
fi

exec "$@"
It does work for all the cases, except running standalone job.
The problem, the way I understand it, is a racing condition.

In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh

tries to start a job immediately once the cluster is created (before connection is established).

Is there a better option to implement it starting a job on container startup?
--

Konstantin Knauf | Solutions Architect

+49 160 91394525
--------------------------

Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra
--

Konstantin Knauf | Solutions Architect

+49 160 91394525

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Data Artisans GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen