Error on deploying Flink docker image with Kubernetes (minikube) and automatically launch a stream WordCount job.
Locked
Error on deploying Flink docker image with Kubernetes (minikube) and automatically launch a stream WordCount job.
 
|
Hi community,
I am trying to deploy the default WordCount stream application on minikube using the official documentation at [1]. I am using minikube v1.13.0 on Ubuntu 18.04 and Kubernetes v1.19.0 on Docker 19.03.8. I could sucessfully start 1 job manager and 3 task managers using the yaml files flink-configuration-configmap.yaml, jobmanager-service.yaml, jobmanager-rest-service.yaml, jobmanager-session-deployment.yaml, and taskmanager-session-deployment.yaml (all available on the Apendix of this link [1]). Then I would like to start the word-count stream job available on the flink jar image [2], which I believe is available since it is built inside the default flink jar distribution. What I understood that I have to do is to create the objects based on the files jobmanager-job.yaml and taskmanager-job-deployment.yaml (also available on the link [1]). And, I think that I have to replace this line below on the object jobmanager-job.yaml (spec.template.spec.containers.name[jobmanager]): args: ["standalone-job", "--job-classname", "org.apache.flink.streaming.examples.wordcount.WordCount"] Is this correct? I am not sure if this is my entire error. I am getting the message "Could not find the provided job class (org.apache.flink.streaming.examples.wordcount.WordCount) in the user lib directory (/opt/flink/usrlib)". As far as I know "/opt/flink/usrlib" is the default directory. I am not sure if I have to change the property: path /host/path/to/job/artifacts. This is my log message of the pod error. Do you guys have any idea of what I am missing in my configuration? Thanks, Felipe $ kubectl get pods NAME READY STATUS RESTARTS AGE flink-jobmanager-ftgg9 0/1 CrashLoopBackOff 3 83s $ kubectl logs flink-jobmanager-ftgg9 Starting Job Manager sed: couldn't open temporary file /opt/flink/conf/sedA699Jt: Read-only file system sed: couldn't open temporary file /opt/flink/conf/sedvZhs0w: Read-only file system /docker-entrypoint.sh: 72: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml: Permission denied /docker-entrypoint.sh: 91: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system Starting standalonejob as a console application on host flink-jobmanager-ftgg9. 2020-09-18 09:15:58,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 2020-09-18 09:15:58,804 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Preconfiguration: 2020-09-18 09:15:58,804 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JM_RESOURCE_PARAMS extraction logs: jvm_params: -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 logs: INFO [] - Loading configuration property: jobmanager.rpc.address, flink-jobmanager INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 4 INFO [] - Loading configuration property: blob.server.port, 6124 INFO [] - Loading configuration property: jobmanager.rpc.port, 6123 INFO [] - Loading configuration property: taskmanager.rpc.port, 6122 INFO [] - Loading configuration property: queryable-state.proxy.ports, 6125 INFO [] - Loading configuration property: jobmanager.memory.process.size, 1600m INFO [] - Loading configuration property: taskmanager.memory.process.size, 1728m INFO [] - Loading configuration property: parallelism.default, 2 INFO [] - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead INFO [] - Final Master Memory configuration: INFO [] - Total Process Memory: 1.563gb (1677721600 bytes) INFO [] - Total Flink Memory: 1.125gb (1207959552 bytes) INFO [] - JVM Heap: 1024.000mb (1073741824 bytes) INFO [] - Off-heap: 128.000mb (134217728 bytes) INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) INFO [] - JVM Overhead: 192.000mb (201326592 bytes) 2020-09-18 09:15:58,805 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 2020-09-18 09:15:58,805 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting StandaloneApplicationClusterEntryPoint (Version: 1.11.0, Scala: 2.11, Rev:d04872d, Date:2020-06-29T16:13:14+02:00) 2020-09-18 09:15:58,805 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user: flink 2020-09-18 09:15:58,805 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current Hadoop/Kerberos user: <no hadoop dependency found> 2020-09-18 09:15:58,805 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.262-b10 2020-09-18 09:15:58,806 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size: 989 MiBytes 2020-09-18 09:15:58,806 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME: /usr/local/openjdk-8 2020-09-18 09:15:58,806 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop Dependency available 2020-09-18 09:15:58,806 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options: 2020-09-18 09:15:58,806 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx1073741824 2020-09-18 09:15:58,806 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms1073741824 2020-09-18 09:15:58,807 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -XX:MaxMetaspaceSize=268435456 2020-09-18 09:15:58,807 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog.file=/opt/flink/log/flink--standalonejob-0-flink-jobmanager-ftgg9.log 2020-09-18 09:15:58,807 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties 2020-09-18 09:15:58,807 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties 2020-09-18 09:15:58,807 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml 2020-09-18 09:15:58,807 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments: 2020-09-18 09:15:58,807 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --configDir 2020-09-18 09:15:58,808 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - /opt/flink/conf 2020-09-18 09:15:58,808 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --job-classname 2020-09-18 09:15:58,808 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - org.apache.flink.streaming.examples.wordcount.WordCount 2020-09-18 09:15:58,808 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath: /opt/flink/lib/flink-csv-1.11.0.jar:/opt/flink/lib/flink-json-1.11.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.11-1.11.0.jar:/opt/flink/lib/flink-table_2.11-1.11.0.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/flink-dist_2.11-1.11.0.jar::: 2020-09-18 09:15:58,808 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 2020-09-18 09:15:58,809 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX signal handlers for [TERM, HUP, INT] 2020-09-18 09:15:58,849 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Could not create application program. org.apache.flink.util.FlinkException: Could not find the provided job class (org.apache.flink.streaming.examples.wordcount.WordCount) in the user lib directory (/opt/flink/usrlib). at org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.getJobClassNameOrScanClassPath(ClassPathPackagedProgramRetriever.java:140) ~[flink-dist_2.11-1.11.0.jar:1.11.0] at org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.getPackagedProgram(ClassPathPackagedProgramRetriever.java:123) ~[flink-dist_2.11-1.11.0.jar:1.11.0] at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:110) ~[flink-dist_2.11-1.11.0.jar:1.11.0] at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:78) [flink-dist_2.11-1.11.0.jar:1.11.0] [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html [2] https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/wordcount/WordCount.java -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com |
|
|
Hi Felipe, Currently, if you want to deploy a standalone job/application Flink cluster on K8s via yamls. You should have your own image with user jar baked located at /opt/flink/usrlib. It could not be specified via config option. Usually, you could add new layer on the official docker image to build in the user jar. A more graceful solution is init container[1], which could download the user jar from remote storage or just copy from local directory /opt/flink/opt to /opt/flink/usrlib. Best, Yang Felipe Gutierrez <[hidden email]> 于2020年9月18日周五 下午5:52写道： Hi community, |
Re: Error on deploying Flink docker image with Kubernetes (minikube) and automatically launch a stream WordCount job.
|
|
thanks Yang,
I got to put it to work in the way that you said. https://github.com/felipegutierrez/explore-flink Best, Felipe -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com On Thu, Sep 24, 2020 at 6:59 AM Yang Wang <[hidden email]> wrote: > > Hi Felipe, > > Currently, if you want to deploy a standalone job/application Flink cluster on K8s via yamls. > You should have your own image with user jar baked located at /opt/flink/usrlib. It could not be > specified via config option. Usually, you could add new layer on the official docker image to > build in the user jar. > > A more graceful solution is init container[1], which could download the user jar from remote > storage or just copy from local directory /opt/flink/opt to /opt/flink/usrlib. > > [1]. https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-initialization/ > > Best, > Yang > > > Felipe Gutierrez <[hidden email]> 于2020年9月18日周五 下午5:52写道： >> >> Hi community, >> >> I am trying to deploy the default WordCount stream application on >> minikube using the official documentation at [1]. I am using minikube >> v1.13.0 on Ubuntu 18.04 and Kubernetes v1.19.0 on Docker 19.03.8. I >> could sucessfully start 1 job manager and 3 task managers using the >> yaml files flink-configuration-configmap.yaml, >> jobmanager-service.yaml, jobmanager-rest-service.yaml, >> jobmanager-session-deployment.yaml, and >> taskmanager-session-deployment.yaml (all available on the Apendix of >> this link [1]). >> >> Then I would like to start the word-count stream job available on the >> flink jar image [2], which I believe is available since it is built >> inside the default flink jar distribution. What I understood that I >> have to do is to create the objects based on the files >> jobmanager-job.yaml and taskmanager-job-deployment.yaml (also >> available on the link [1]). And, I think that I have to replace this >> line below on the object jobmanager-job.yaml >> (spec.template.spec.containers.name[jobmanager]): >> >> args: ["standalone-job", "--job-classname", >> "org.apache.flink.streaming.examples.wordcount.WordCount"] >> >> Is this correct? I am not sure if this is my entire error. I am >> getting the message "Could not find the provided job class >> (org.apache.flink.streaming.examples.wordcount.WordCount) in the user >> lib directory (/opt/flink/usrlib)". As far as I know >> "/opt/flink/usrlib" is the default directory. I am not sure if I have >> to change the property: path /host/path/to/job/artifacts. This is my >> log message of the pod error. >> Do you guys have any idea of what I am missing in my configuration? >> >> Thanks, Felipe >> >> $ kubectl get pods >> NAME READY STATUS RESTARTS AGE >> flink-jobmanager-ftgg9 0/1 CrashLoopBackOff 3 83s >> $ kubectl logs flink-jobmanager-ftgg9 >> Starting Job Manager >> sed: couldn't open temporary file /opt/flink/conf/sedA699Jt: Read-only >> file system >> sed: couldn't open temporary file /opt/flink/conf/sedvZhs0w: Read-only >> file system >> /docker-entrypoint.sh: 72: /docker-entrypoint.sh: cannot create >> /opt/flink/conf/flink-conf.yaml: Permission denied >> /docker-entrypoint.sh: 91: /docker-entrypoint.sh: cannot create >> /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system >> Starting standalonejob as a console application on host flink-jobmanager-ftgg9. >> 2020-09-18 09:15:58,801 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -------------------------------------------------------------------------------- >> 2020-09-18 09:15:58,804 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Preconfiguration: >> 2020-09-18 09:15:58,804 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> >> >> JM_RESOURCE_PARAMS extraction logs: >> jvm_params: -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 >> logs: INFO [] - Loading configuration property: >> jobmanager.rpc.address, flink-jobmanager >> INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 4 >> INFO [] - Loading configuration property: blob.server.port, 6124 >> INFO [] - Loading configuration property: jobmanager.rpc.port, 6123 >> INFO [] - Loading configuration property: taskmanager.rpc.port, 6122 >> INFO [] - Loading configuration property: queryable-state.proxy.ports, 6125 >> INFO [] - Loading configuration property: jobmanager.memory.process.size, 1600m >> INFO [] - Loading configuration property: >> taskmanager.memory.process.size, 1728m >> INFO [] - Loading configuration property: parallelism.default, 2 >> INFO [] - The derived from fraction jvm overhead memory (160.000mb >> (167772162 bytes)) is less than its min value 192.000mb (201326592 >> bytes), min value will be used instead >> INFO [] - Final Master Memory configuration: >> INFO [] - Total Process Memory: 1.563gb (1677721600 bytes) >> INFO [] - Total Flink Memory: 1.125gb (1207959552 bytes) >> INFO [] - JVM Heap: 1024.000mb (1073741824 bytes) >> INFO [] - Off-heap: 128.000mb (134217728 bytes) >> INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) >> INFO [] - JVM Overhead: 192.000mb (201326592 bytes) >> >> 2020-09-18 09:15:58,805 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -------------------------------------------------------------------------------- >> 2020-09-18 09:15:58,805 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Starting StandaloneApplicationClusterEntryPoint (Version: 1.11.0, >> Scala: 2.11, Rev:d04872d, Date:2020-06-29T16:13:14+02:00) >> 2020-09-18 09:15:58,805 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS >> current user: flink >> 2020-09-18 09:15:58,805 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Current Hadoop/Kerberos user: <no hadoop dependency found> >> 2020-09-18 09:15:58,805 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.262-b10 >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Maximum heap size: 989 MiBytes >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> JAVA_HOME: /usr/local/openjdk-8 >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No >> Hadoop Dependency available >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM >> Options: >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Xmx1073741824 >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Xms1073741824 >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -XX:MaxMetaspaceSize=268435456 >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Dlog.file=/opt/flink/log/flink--standalonejob-0-flink-jobmanager-ftgg9.log >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Program Arguments: >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> --configDir >> 2020-09-18 09:15:58,808 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> /opt/flink/conf >> 2020-09-18 09:15:58,808 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> --job-classname >> 2020-09-18 09:15:58,808 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> org.apache.flink.streaming.examples.wordcount.WordCount >> 2020-09-18 09:15:58,808 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Classpath: /opt/flink/lib/flink-csv-1.11.0.jar:/opt/flink/lib/flink-json-1.11.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.11-1.11.0.jar:/opt/flink/lib/flink-table_2.11-1.11.0.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/flink-dist_2.11-1.11.0.jar::: >> 2020-09-18 09:15:58,808 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -------------------------------------------------------------------------------- >> 2020-09-18 09:15:58,809 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Registered UNIX signal handlers for [TERM, HUP, INT] >> 2020-09-18 09:15:58,849 ERROR >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Could not create application program. >> org.apache.flink.util.FlinkException: Could not find the provided job >> class (org.apache.flink.streaming.examples.wordcount.WordCount) in the >> user lib directory (/opt/flink/usrlib). >> at org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.getJobClassNameOrScanClassPath(ClassPathPackagedProgramRetriever.java:140) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.getPackagedProgram(ClassPathPackagedProgramRetriever.java:123) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:110) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:78) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> >> [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html >> [2] https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/wordcount/WordCount.java >> -- >> -- Felipe Gutierrez >> -- skype: felipe.o.gutierrez >> -- https://felipeogutierrez.blogspot.com |
|
|
Glad to hear that. And the repository you shared is very helpful for the users who are trying to deploy Flink clusters on K8s. Best, Yang Felipe Gutierrez <[hidden email]> 于2020年9月24日周四 下午2:04写道： thanks Yang, |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |