Fail to deploy Flink on minikube
Locked
Fail to deploy Flink on minikube
 
|
Re: Fail to deploy Flink on minikube
|
|
Re: Fail to deploy Flink on minikube
|
|
Re: Fail to deploy Flink on minikube
|
|
Re: Fail to deploy Flink on minikube
|
|
Re: Fail to deploy Flink on minikube
|
|
Hi Till, This is the taskManager log As you see, the logs print ‘line 92 -- Could not connect to flink-jobmanager:6123’ then print ‘line 128 --Could not resolve ResourceManager address akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*.’ And repeat print this A few minutes later, the taskmanger shut down and restart This is my yaml files, could u help me to confirm did I omitted something? Thanks a lot! --------------------------------------------------- flink-configuration-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: flink-config labels: app: flink data: flink-conf.yaml: |+ jobmanager.rpc.address: flink-jobmanager taskmanager.numberOfTaskSlots: 1 blob.server.port: 6124 jobmanager.rpc.port: 6123 taskmanager.rpc.port: 6122 queryable-state.proxy.ports: 6125 jobmanager.memory.process.size: 1024m taskmanager.memory.process.size: 1024m parallelism.default: 1 log4j-console.properties: |+ rootLogger.level = INFO rootLogger.appenderRef.console.ref = ConsoleAppender rootLogger.appenderRef.rolling.ref = RollingFileAppender logger.akka.name = akka logger.akka.level = INFO logger.kafka.name= org.apache.kafka logger.kafka.level = INFO logger.hadoop.name = org.apache.hadoop logger.hadoop.level = INFO logger.zookeeper.name = org.apache.zookeeper logger.zookeeper.level = INFO appender.console.name = ConsoleAppender appender.console.type = CONSOLE appender.console.layout.type = PatternLayout appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n appender.rolling.name = RollingFileAppender appender.rolling.type = RollingFile appender.rolling.append = false appender.rolling.fileName = ${sys:log.file} appender.rolling.filePattern = ${sys:log.file}.%i appender.rolling.layout.type = PatternLayout appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n appender.rolling.policies.type = Policies appender.rolling.policies.size.type = SizeBasedTriggeringPolicy appender.rolling.policies.size.size=100MB appender.rolling.strategy.type = DefaultRolloverStrategy appender.rolling.strategy.max = 10 logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline logger.netty.level = OFF --------------------------------------------------- jobmanager-service.yaml apiVersion: v1 kind: Service metadata: name: flink-jobmanager spec: type: ClusterIP ports: - name: rpc port: 6123 - name: blob-server port: 6124 - name: webui port: 8081 selector: app: flink component: jobmanager -------------------------------------------------- jobmanager-session-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: flink-jobmanager spec: replicas: 1 selector: matchLabels: app: flink component: jobmanager template: metadata: labels: app: flink component: jobmanager spec: containers: - name: jobmanager image: registry.cn-hangzhou.aliyuncs.com/superainbower/flink:1.11.1 args: ["jobmanager"] ports: - containerPort: 6123 name: rpc - containerPort: 6124 name: blob-server - containerPort: 8081 name: webui livenessProbe: tcpSocket: port: 6123 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties imagePullSecrets: - name: regcred --------------------------------------------------- taskmanager-session-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: flink-taskmanager spec: replicas: 1 selector: matchLabels: app: flink component: taskmanager template: metadata: labels: app: flink component: taskmanager spec: containers: - name: taskmanager image: registry.cn-hangzhou.aliyuncs.com/superainbower/flink:1.11.1 args: ["taskmanager"] ports: - containerPort: 6122 name: rpc - containerPort: 6125 name: query-state livenessProbe: tcpSocket: port: 6122 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf/ securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties imagePullSecrets: - name: regcred On 09/2/2020 20:38，[hidden email] wrote：
|
taskmanger.log (28K)
Re: Fail to deploy Flink on minikube
|
|
Hi Till, I find something may be helpful. The kubernetes Dashboard show job-manager ip 172.18.0.5, task-manager ip 172.18.0.6 When I run command 'kubectl exec -ti flink-taskmanager-74c68c6f48-jqpbn -- /bin/bash’ && ‘ping 172.18.0.5’ I can get response But when I ping flink-jobmanager ,there is no response
On 09/3/2020 09:03，[hidden email] wrote：
|
Re: Fail to deploy Flink on minikube
|
|
I guess something is wrong with your kube proxy, which causes TaskManager could not connect to JobManager. You could verify this by directly using JobManager Pod ip instead of service name. Please do as follows. * Edit the TaskManager deployment(via kubectl edit flink-taskmanager) and update the args field to the following. args: ["taskmanager", "-Djobmanager.rpc.address=172.18.0.5"] Given that "172.18.0.5" is the JobManager pod ip. * Delete the current TaskManager pod and let restart again * Now check the TaskManager logs to check whether it could register successfully Best, Yang superainbower <[hidden email]> 于2020年9月3日周四 上午9:35写道：
|
Re: Fail to deploy Flink on minikube
|
|
HI Yang, I update taskmanager-session-deployment.yaml like this: apiVersion: apps/v1 kind: Deployment metadata: name: flink-taskmanager spec: replicas: 1 selector: matchLabels: app: flink component: taskmanager template: metadata: labels: app: flink component: taskmanager spec: containers: - name: taskmanager image: registry.cn-hangzhou.aliyuncs.com/superainbower/flink:1.11.1 args: ["taskmanager","-Djobmanager.rpc.address=172.18.0.5"] ports: - containerPort: 6122 name: rpc - containerPort: 6125 name: query-state livenessProbe: tcpSocket: port: 6122 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf/ securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties imagePullSecrets: - name: regcred And Delete the TaskManager pod and restart it , but the logs print this Could not resolve ResourceManager address akka.tcp://flink@172.18.0.5:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@172.18.0.5:6123/user/rpc/resourcemanager_*
It change flink-jobmanager to 172.18.0.5
On 09/3/2020 11:09，[hidden email] wrote：
|
Re: Fail to deploy Flink on minikube
|
|
Sorry i forget that the JobManager is binding its rpc address to flink-jobmanager, not the ip address. So you need to also update the jobmanager-session-deployment.yaml with following changes. ... containers: - name: jobmanager env: - name: JM_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP image: flink:1.11 args: ["jobmanager", "$(JM_IP)"] ... After then the JobManager is binding the rpc address with its ip. Best, Yang superainbower <[hidden email]> 于2020年9月3日周四 上午11:38写道：
|
Re: Fail to deploy Flink on minikube
|
|
In order to exclude a Minikube problem, you could also try to run Flink on an older Minikube and an older K8s version. Our end-to-end tests use Minikube v1.8.2, for example. Cheers, Till On Thu, Sep 3, 2020 at 8:44 AM Yang Wang <[hidden email]> wrote:
|
Re: Fail to deploy Flink on minikube
|
|
Hi Till & Yang, I can deploy Flink on kubernetes(not minikube), it works well So there are some problem about my minikube but I can’t find and fix it Anyway I can deploy on k8s now Thanks for your help! On 09/3/2020 15:47，[hidden email] wrote：
|
Re: Fail to deploy Flink on minikube
|
|
Great to hear that it works on K8s and letting us know that the problem is likely to be caused by Minikube. Cheers, Till On Fri, Sep 4, 2020 at 8:53 AM superainbower <[hidden email]> wrote:
|
| Free forum by Nabble | Edit this page |