Kubernetes Application Cluster Not Working

[Claude Murad]

Hello, 

I'm trying to setup Flink in Kubernetes using the Application Mode as described here: https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes

The doc mentions that there needs to be a aervice exposing the JobManager’s REST and UI ports.  It then points to a link w/ the resource definitions: https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#application-cluster-resource-definitions and I defined the following service along w/ the jobmanager, taskmanager, and flink-conf. 

apiVersion: v1
kind: Service
metadata:
  name: flink-jobmanager
spec:
  type: ClusterIP
  ports:
  - name: rpc
    port: 6123
  - name: blob-server
    port: 6124
  - name: webui
    port: 8081
  selector:
    app: flink
    component: jobmanager

I am able to access the jobmanager UI but the taskmanagers are failing w/ the following error:

Could not resolve ResourceManager address akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*

Any ideas about this?  

Thanks

[Guowei Ma]

Hi, M

Could you give the full stack? This might not be the root cause.

On Wed, Mar 24, 2021 at 2:46 AM Claude M <[hidden email]> wrote:

Hello, 

I'm trying to setup Flink in Kubernetes using the Application Mode as described here: https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes

The doc mentions that there needs to be a aervice exposing the JobManager’s REST and UI ports.  It then points to a link w/ the resource definitions: https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#application-cluster-resource-definitions and I defined the following service along w/ the jobmanager, taskmanager, and flink-conf. 

apiVersion: v1
kind: Service
metadata:
  name: flink-jobmanager
spec:
  type: ClusterIP
  ports:
  - name: rpc
    port: 6123
  - name: blob-server
    port: 6124
  - name: webui
    port: 8081
  selector:
    app: flink
    component: jobmanager

I am able to access the jobmanager UI but the taskmanagers are failing w/ the following error:

Could not resolve ResourceManager address akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*

Any ideas about this?  

Thanks

[Yang Wang]

Are you sure that the JobManager akka address is binded to "flink-jobmanager"?

You could set "jobmanager.rpc.address" to flink-jobmanager in the ConfigMap.

Best,

Yang

Guowei Ma <[hidden email]> 于2021年3月24日周三 上午10:22写道：

Hi, M

Could you give the full stack? This might not be the root cause.

Best,

Guowei

On Wed, Mar 24, 2021 at 2:46 AM Claude M <[hidden email]> wrote:

Hello, 

I'm trying to setup Flink in Kubernetes using the Application Mode as described here: https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes

The doc mentions that there needs to be a aervice exposing the JobManager’s REST and UI ports.  It then points to a link w/ the resource definitions: https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#application-cluster-resource-definitions and I defined the following service along w/ the jobmanager, taskmanager, and flink-conf. 

apiVersion: v1
kind: Service
metadata:
  name: flink-jobmanager
spec:
  type: ClusterIP
  ports:
  - name: rpc
    port: 6123
  - name: blob-server
    port: 6124
  - name: webui
    port: 8081
  selector:
    app: flink
    component: jobmanager

I am able to access the jobmanager UI but the taskmanagers are failing w/ the following error:

Could not resolve ResourceManager address akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*

Any ideas about this?  

Thanks

[Claude Murad]

This issue was resolved by adding the following environment variable to both the jobmanager and taskmanager:

- name: JOB_MANAGER_RPC_ADDRESS
  value: jobmanager

On Wed, Mar 24, 2021 at 1:33 AM Yang Wang <[hidden email]> wrote:

Are you sure that the JobManager akka address is binded to "flink-jobmanager"?

You could set "jobmanager.rpc.address" to flink-jobmanager in the ConfigMap.

Best,

Yang

Guowei Ma <[hidden email]> 于2021年3月24日周三 上午10:22写道：

Hi, M

Could you give the full stack? This might not be the root cause.

Best,

Guowei

On Wed, Mar 24, 2021 at 2:46 AM Claude M <[hidden email]> wrote:

Hello, 

I'm trying to setup Flink in Kubernetes using the Application Mode as described here: https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes

The doc mentions that there needs to be a aervice exposing the JobManager’s REST and UI ports.  It then points to a link w/ the resource definitions: https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#application-cluster-resource-definitions and I defined the following service along w/ the jobmanager, taskmanager, and flink-conf. 

apiVersion: v1
kind: Service
metadata:
  name: flink-jobmanager
spec:
  type: ClusterIP
  ports:
  - name: rpc
    port: 6123
  - name: blob-server
    port: 6124
  - name: webui
    port: 8081
  selector:
    app: flink
    component: jobmanager

I am able to access the jobmanager UI but the taskmanagers are failing w/ the following error:

Could not resolve ResourceManager address akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*

Any ideas about this?  

Thanks