Flink Job Cluster Deployment on K8s

Posted by Thad Truman on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-Job-Cluster-Deployment-on-K8s-tp23953.html

Hello,

 

I am trying to experiment with the new Flink job cluster on Kubernetes that is available with the Flink 1.6.x release.

 

I am using the instructions here to create the docker image, which is working great.   This image then gets pushed to our Artifactory.

 

I am able to create the job cluster service using this helm chart.

 

However when I try to deploy the job cluster job using the helm chart below (based on this one):

 

apiVersion: batch/v1
kind: Job
metadata:
  name: flink-job-cluster
spec:
  template:
    metadata:
      labels:
        app: flink
        component: job-cluster
    spec:
      imagePullSecrets:
      - name: artifactory-docker-registry
      restartPolicy: OnFailure
      containers:
      - name: flink-job-cluster
        image: {ImageOnArtifactory}
        args: ["job-cluster", "--job-classname", "job.jar", "-Djobmanager.rpc.address=flink-job-cluster",
               "-Dparallelism.default=1", "-Dblob.server.port=6124", "-Dquery.server.ports=6125"]
        ports:
        - containerPort: 6123
          name: rpc
        - containerPort: 6124
          name: blob
        - containerPort: 6125
          name: query
        - containerPort: 8081
          name: ui

 

I get this error on the pod:

 

%d [%thread] %-5level %logger - %msg%n org.apache.flink.util.FlinkException: Could not load the provied entrypoint class.                                                                                               

at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:102) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                       

at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.retrieveJobGraph(StandaloneJobClusterEntryPoint.java:84) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                             

at org.apache.flink.runtime.entrypoint.JobClusterEntrypoint.createDispatcher(JobClusterEntrypoint.java:107) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                                                   

at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startClusterComponents(ClusterEntrypoint.java:353) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                                                  

at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:232) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                                                              

at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:190) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                                                   

at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                                                             

at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:189) [flink-dist_2.11-1.6.1.jar:1.6.1]                                                                              

at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:170) [flink-dist_2.11-1.6.1.jar:1.6.1]

Caused by: java.lang.ClassNotFoundException: job.jar                                                                                                                                                                    

at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_111-internal]                                                                                                                           

at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_111-internal]                                                                                                                                 

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_111-internal]                                                                                                                         

at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_111-internal]                                                                                                                                 

at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:99) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                       

... 8 more

 

 

When I launch and attach the docker image, job.jar exists in /opt/flink and there is a symbolic link in /opt/flink/lib.

 

Any ideas as to why job.jar can’t be found?

 

Our flink version is 1.6.1.  We are using the flink-1.6.1-bin-scala_2.11 distribution.

 

Any help would be much appreciated.

 

Thanks,

 

Thad Truman | Software Engineer | Neovest, Inc. A: T: E: 1145 S 800 E, Ste 310 Orem, UT 84097 [phone] [hidden email]   Support Desk: [hidden email] / [phone]
This email is confidential and subject to important disclaimers and conditions including on offers for purchase or sale of securities accuracy and completeness of information viruses confidentiality legal privilege and legal entity disclaimers available at www.neovest.com/disclosures.html