Kubernetes HA Services - artifact for KubernetesHaServicesFactory

[Ashish Nigam]

Hi,

I am not able to identify maven artifact that will have implementation for this class 

org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory

I am using info in this link to test out HA implementation

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/ha/kubernetes_ha.html

Please point me to the right artifact. Also, I plan to use S3 bucket as storageDir. So, will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Thanks

Ashish

[Yang Wang]

Class "org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory" is introduced from 1.12.0 version.

You could try with the latest version 1.12.1[1].

Will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Yes, you need to copy the s3 fs implementation to plugin directory.

An alternative is you could set the environment to enable the plugin[1].

[1]. https://hub.docker.com/r/apache/flink/tags?page=1&ordering=last_updated&name=1.12.1

[2]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/standalone/docker.html#using-filesystem-plugins

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月18日周一 下午11:15写道：

Hi,

I am not able to identify maven artifact that will have implementation for this class 

org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory

I am using info in this link to test out HA implementation

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/ha/kubernetes_ha.html

Please point me to the right artifact. Also, I plan to use S3 bucket as storageDir. So, will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Thanks

Ashish

[Ashish Nigam]

I did move to 1.12.1 version and also ensured that docker has kubernetes jar file in the right location, i.e.  /opt/flink/plugins/s3-fs-presto/flink-kubernetes_2.12-1.12.1.jar

But job manager is still not able to find the class

2021-01-21 00:00:49,376 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Could not start cluster entrypoint StandaloneApplicationClusterEntryPoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneApplicationClusterEntryPoint.
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:520) [flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:94) [flink-dist_2.12-1.11.2.jar:1.11.2]
Caused by: org.apache.flink.util.FlinkException: Could not instantiate class 'org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory' of type 'org.apache.flink.runtime.highavailability.HighAvailabilityServicesFactory'. Please make sure that this class is on your class path.
at org.apache.flink.util.InstantiationUtil.instantiate(InstantiationUtil.java:352) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.loadCustomHighAvailabilityServicesFactory(HighAvailabilityServicesUtils.java:263) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:246) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:126) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:306) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:269) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:211) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
... 2 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]

On Mon, Jan 18, 2021 at 7:52 PM Yang Wang <[hidden email]> wrote:

Class "org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory" is introduced from 1.12.0 version.

You could try with the latest version 1.12.1[1].

Will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Yes, you need to copy the s3 fs implementation to plugin directory.

An alternative is you could set the environment to enable the plugin[1].

[1]. https://hub.docker.com/r/apache/flink/tags?page=1&ordering=last_updated&name=1.12.1

[2]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/standalone/docker.html#using-filesystem-plugins

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月18日周一 下午11:15写道：

Hi,

I am not able to identify maven artifact that will have implementation for this class 

org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory

I am using info in this link to test out HA implementation

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/ha/kubernetes_ha.html

Please point me to the right artifact. Also, I plan to use S3 bucket as storageDir. So, will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Thanks

Ashish

[Yang Wang]

You do not need to put flink-kubernetes_2.12-1.12.1.jar under the plugin directory. Only the S3 fs jar

needs to be put there.

I think I found the root cause. It seems your flink-dist is still 1.11.

Do you want to use the image 1.11 and enable the K8s HA at the same time?

I think it could not work. Because the HA related codes have been refactored from release 1.12.

Even though you copy the flink-kubernetes_2.12-1.12.1.jar to /opt/flink/lib and make the KubernetesHaServicesFactory class 

could be resolved. It will encounter other issues after then.

Could you please try with the latest Flink 1.12 image?

docker pull apache/flink:1.12.1

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月21日周四 上午8:05写道：

I did move to 1.12.1 version and also ensured that docker has kubernetes jar file in the right location, i.e.  /opt/flink/plugins/s3-fs-presto/flink-kubernetes_2.12-1.12.1.jar

But job manager is still not able to find the class

2021-01-21 00:00:49,376 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Could not start cluster entrypoint StandaloneApplicationClusterEntryPoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneApplicationClusterEntryPoint.
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:520) [flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:94) [flink-dist_2.12-1.11.2.jar:1.11.2]
Caused by: org.apache.flink.util.FlinkException: Could not instantiate class 'org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory' of type 'org.apache.flink.runtime.highavailability.HighAvailabilityServicesFactory'. Please make sure that this class is on your class path.
at org.apache.flink.util.InstantiationUtil.instantiate(InstantiationUtil.java:352) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.loadCustomHighAvailabilityServicesFactory(HighAvailabilityServicesUtils.java:263) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:246) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:126) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:306) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:269) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:211) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
... 2 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]

On Mon, Jan 18, 2021 at 7:52 PM Yang Wang <[hidden email]> wrote:

Class "org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory" is introduced from 1.12.0 version.

You could try with the latest version 1.12.1[1].

Will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Yes, you need to copy the s3 fs implementation to plugin directory.

An alternative is you could set the environment to enable the plugin[1].

[1]. https://hub.docker.com/r/apache/flink/tags?page=1&ordering=last_updated&name=1.12.1

[2]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/standalone/docker.html#using-filesystem-plugins

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月18日周一 下午11:15写道：

Hi,

I am not able to identify maven artifact that will have implementation for this class 

org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory

I am using info in this link to test out HA implementation

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/ha/kubernetes_ha.html

Please point me to the right artifact. Also, I plan to use S3 bucket as storageDir. So, will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Thanks

Ashish

[Ashish Nigam]

It works now. Job manager is able to start. But now, I have run into another issue.

It seems job manager is trying to create configmap in default namespace and namespace/service account where I run job manager does not have access to configmap

GET at: https://X.X.X.X/api/v1/namespaces/default/configmaps/dev-cluster-restserver-leader. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps "dev-cluster-restserver-leader" is forbidden: User "system:serviceaccount:apps-dev:apps" cannot get resource "configmaps" in API group "" in the namespace "default".

Is there a workaround apart from adding permission? Can leader configmap be created in the same namespace as Job Manager?

On Wed, Jan 20, 2021 at 7:42 PM Yang Wang <[hidden email]> wrote:

You do not need to put flink-kubernetes_2.12-1.12.1.jar under the plugin directory. Only the S3 fs jar

needs to be put there.

I think I found the root cause. It seems your flink-dist is still 1.11.

Do you want to use the image 1.11 and enable the K8s HA at the same time?

I think it could not work. Because the HA related codes have been refactored from release 1.12.

Even though you copy the flink-kubernetes_2.12-1.12.1.jar to /opt/flink/lib and make the KubernetesHaServicesFactory class 

could be resolved. It will encounter other issues after then.

Could you please try with the latest Flink 1.12 image?

docker pull apache/flink:1.12.1

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月21日周四 上午8:05写道：

I did move to 1.12.1 version and also ensured that docker has kubernetes jar file in the right location, i.e.  /opt/flink/plugins/s3-fs-presto/flink-kubernetes_2.12-1.12.1.jar

But job manager is still not able to find the class

2021-01-21 00:00:49,376 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Could not start cluster entrypoint StandaloneApplicationClusterEntryPoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneApplicationClusterEntryPoint.
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:520) [flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:94) [flink-dist_2.12-1.11.2.jar:1.11.2]
Caused by: org.apache.flink.util.FlinkException: Could not instantiate class 'org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory' of type 'org.apache.flink.runtime.highavailability.HighAvailabilityServicesFactory'. Please make sure that this class is on your class path.
at org.apache.flink.util.InstantiationUtil.instantiate(InstantiationUtil.java:352) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.loadCustomHighAvailabilityServicesFactory(HighAvailabilityServicesUtils.java:263) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:246) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:126) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:306) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:269) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:211) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
... 2 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]

On Mon, Jan 18, 2021 at 7:52 PM Yang Wang <[hidden email]> wrote:

Class "org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory" is introduced from 1.12.0 version.

You could try with the latest version 1.12.1[1].

Will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Yes, you need to copy the s3 fs implementation to plugin directory.

An alternative is you could set the environment to enable the plugin[1].

[1]. https://hub.docker.com/r/apache/flink/tags?page=1&ordering=last_updated&name=1.12.1

[2]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/standalone/docker.html#using-filesystem-plugins

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月18日周一 下午11:15写道：

Hi,

I am not able to identify maven artifact that will have implementation for this class 

org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory

I am using info in this link to test out HA implementation

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/ha/kubernetes_ha.html

Please point me to the right artifact. Also, I plan to use S3 bucket as storageDir. So, will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Thanks

Ashish

[Yang Wang]

You could set config option "kubernetes.namespace" to your flink-conf ConfigMap. And then 

KubernetesHAService will use it to create/watch the ConfigMap. Please note the default service

account has enough permission.

Of course, you could also set the config option "kubernetes.service-account" to another service account

with enough permission.

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月22日周五 上午3:16写道：

It works now. Job manager is able to start. But now, I have run into another issue.

It seems job manager is trying to create configmap in default namespace and namespace/service account where I run job manager does not have access to configmap

GET at: https://X.X.X.X/api/v1/namespaces/default/configmaps/dev-cluster-restserver-leader. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps "dev-cluster-restserver-leader" is forbidden: User "system:serviceaccount:apps-dev:apps" cannot get resource "configmaps" in API group "" in the namespace "default".

Is there a workaround apart from adding permission? Can leader configmap be created in the same namespace as Job Manager?

On Wed, Jan 20, 2021 at 7:42 PM Yang Wang <[hidden email]> wrote:

You do not need to put flink-kubernetes_2.12-1.12.1.jar under the plugin directory. Only the S3 fs jar

needs to be put there.

I think I found the root cause. It seems your flink-dist is still 1.11.

Do you want to use the image 1.11 and enable the K8s HA at the same time?

I think it could not work. Because the HA related codes have been refactored from release 1.12.

Even though you copy the flink-kubernetes_2.12-1.12.1.jar to /opt/flink/lib and make the KubernetesHaServicesFactory class 

could be resolved. It will encounter other issues after then.

Could you please try with the latest Flink 1.12 image?

docker pull apache/flink:1.12.1

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月21日周四 上午8:05写道：

I did move to 1.12.1 version and also ensured that docker has kubernetes jar file in the right location, i.e.  /opt/flink/plugins/s3-fs-presto/flink-kubernetes_2.12-1.12.1.jar

But job manager is still not able to find the class

2021-01-21 00:00:49,376 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Could not start cluster entrypoint StandaloneApplicationClusterEntryPoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneApplicationClusterEntryPoint.
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:520) [flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:94) [flink-dist_2.12-1.11.2.jar:1.11.2]
Caused by: org.apache.flink.util.FlinkException: Could not instantiate class 'org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory' of type 'org.apache.flink.runtime.highavailability.HighAvailabilityServicesFactory'. Please make sure that this class is on your class path.
at org.apache.flink.util.InstantiationUtil.instantiate(InstantiationUtil.java:352) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.loadCustomHighAvailabilityServicesFactory(HighAvailabilityServicesUtils.java:263) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:246) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:126) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:306) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:269) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:211) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
... 2 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]

On Mon, Jan 18, 2021 at 7:52 PM Yang Wang <[hidden email]> wrote:

Class "org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory" is introduced from 1.12.0 version.

You could try with the latest version 1.12.1[1].

Will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Yes, you need to copy the s3 fs implementation to plugin directory.

An alternative is you could set the environment to enable the plugin[1].

[1]. https://hub.docker.com/r/apache/flink/tags?page=1&ordering=last_updated&name=1.12.1

[2]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/standalone/docker.html#using-filesystem-plugins

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月18日周一 下午11:15写道：

Hi,

I am not able to identify maven artifact that will have implementation for this class 

org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory

I am using info in this link to test out HA implementation

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/ha/kubernetes_ha.html

Please point me to the right artifact. Also, I plan to use S3 bucket as storageDir. So, will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Thanks

Ashish

[Ashish Nigam]

Yang, It worked finally... after adding kubernetes related config in configmap. 

On Thu, Jan 21, 2021 at 7:13 PM Yang Wang <[hidden email]> wrote:

You could set config option "kubernetes.namespace" to your flink-conf ConfigMap. And then 

KubernetesHAService will use it to create/watch the ConfigMap. Please note the default service

account has enough permission.

Of course, you could also set the config option "kubernetes.service-account" to another service account

with enough permission.

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月22日周五 上午3:16写道：

It works now. Job manager is able to start. But now, I have run into another issue.

It seems job manager is trying to create configmap in default namespace and namespace/service account where I run job manager does not have access to configmap

GET at: https://X.X.X.X/api/v1/namespaces/default/configmaps/dev-cluster-restserver-leader. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps "dev-cluster-restserver-leader" is forbidden: User "system:serviceaccount:apps-dev:apps" cannot get resource "configmaps" in API group "" in the namespace "default".

Is there a workaround apart from adding permission? Can leader configmap be created in the same namespace as Job Manager?

On Wed, Jan 20, 2021 at 7:42 PM Yang Wang <[hidden email]> wrote:

You do not need to put flink-kubernetes_2.12-1.12.1.jar under the plugin directory. Only the S3 fs jar

needs to be put there.

I think I found the root cause. It seems your flink-dist is still 1.11.

Do you want to use the image 1.11 and enable the K8s HA at the same time?

I think it could not work. Because the HA related codes have been refactored from release 1.12.

Even though you copy the flink-kubernetes_2.12-1.12.1.jar to /opt/flink/lib and make the KubernetesHaServicesFactory class 

could be resolved. It will encounter other issues after then.

Could you please try with the latest Flink 1.12 image?

docker pull apache/flink:1.12.1

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月21日周四 上午8:05写道：

I did move to 1.12.1 version and also ensured that docker has kubernetes jar file in the right location, i.e.  /opt/flink/plugins/s3-fs-presto/flink-kubernetes_2.12-1.12.1.jar

But job manager is still not able to find the class

2021-01-21 00:00:49,376 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Could not start cluster entrypoint StandaloneApplicationClusterEntryPoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneApplicationClusterEntryPoint.
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:520) [flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:94) [flink-dist_2.12-1.11.2.jar:1.11.2]
Caused by: org.apache.flink.util.FlinkException: Could not instantiate class 'org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory' of type 'org.apache.flink.runtime.highavailability.HighAvailabilityServicesFactory'. Please make sure that this class is on your class path.
at org.apache.flink.util.InstantiationUtil.instantiate(InstantiationUtil.java:352) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.loadCustomHighAvailabilityServicesFactory(HighAvailabilityServicesUtils.java:263) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:246) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:126) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:306) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:269) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:211) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
... 2 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]

On Mon, Jan 18, 2021 at 7:52 PM Yang Wang <[hidden email]> wrote:

Class "org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory" is introduced from 1.12.0 version.

You could try with the latest version 1.12.1[1].

Will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Yes, you need to copy the s3 fs implementation to plugin directory.

An alternative is you could set the environment to enable the plugin[1].

[1]. https://hub.docker.com/r/apache/flink/tags?page=1&ordering=last_updated&name=1.12.1

[2]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/standalone/docker.html#using-filesystem-plugins

Best,

Yang

Ashish Nigam <[hidden email]> 于2021年1月18日周一 下午11:15写道：

Hi,

I am not able to identify maven artifact that will have implementation for this class 

org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory

I am using info in this link to test out HA implementation

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/ha/kubernetes_ha.html

Please point me to the right artifact. Also, I plan to use S3 bucket as storageDir. So, will that jar file need to be copied under /opt/flink/plugins/s3-fs-presto folder? 

Thanks

Ashish