Flink TLS in K8s

[Patrick Eifler]

Hi,

I did set up a flink session cluster on K8s. 

Now I added the ssl configuration as shown in the documentation:

# Flink TLS

security.ssl.internal.enabled: true

security.ssl.internal.keystore: /config/internal-keystore/internal.keystore.jks

security.ssl.internal.truststore: /config/internal-keystore/internal.keystore.jks

security.ssl.internal.keystore-password: {{ .Values.keystore.password }}

security.ssl.internal.truststore-password: {{ .Values.keystore.password }}

security.ssl.internal.key-password: {{ .Values.keystore.password }}

Now I get the problem that the task manager cannot connect to the job manager nor the resource manager:

could not resolve ResourceManager address <a href="akka.ssl.tcp://flink@flink-sc-jobmanager:6123/user/rpc/resourcemanager_*" class="">akka.ssl.tcp://flink@flink-sc-jobmanager:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address <a href="akka.ssl.tcp://flink@flink-sc-jobmanager:6123/user/rpc/resourcemanager_*" class="">akka.ssl.tcp://flink@flink-sc-jobmanager:6123/user/rpc/resourcemanager_*.

Do I need to change the job manager port to make this work?

Any suggestions would be highly appreciated.

Thanks.

Patrick

[Chesnay Schepler]

It is unlikely that this is a port issue, and I would currently suspect that something in your SSL setup is not correct.

@Nico: do you have a suggestion on how to debug this?

On 11/5/2020 4:23 PM, Patrick Eifler wrote:

Hi,

I did set up a flink session cluster on K8s. 

Now I added the ssl configuration as shown in the documentation:

# Flink TLS

security.ssl.internal.enabled: true

security.ssl.internal.keystore: /config/internal-keystore/internal.keystore.jks

security.ssl.internal.truststore: /config/internal-keystore/internal.keystore.jks

security.ssl.internal.keystore-password: {{ .Values.keystore.password }}

security.ssl.internal.truststore-password: {{ .Values.keystore.password }}

security.ssl.internal.key-password: {{ .Values.keystore.password }}

Now I get the problem that the task manager cannot connect to the job manager nor the resource manager:

could not resolve ResourceManager address <a href="akka.ssl.tcp://flink@flink-sc-jobmanager:6123/user/rpc/resourcemanager_*" class="" moz-do-not-send="true">akka.ssl.tcp://flink@flink-sc-jobmanager:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address <a href="akka.ssl.tcp://flink@flink-sc-jobmanager:6123/user/rpc/resourcemanager_*" class="" moz-do-not-send="true">akka.ssl.tcp://flink@flink-sc-jobmanager:6123/user/rpc/resourcemanager_*.

Do I need to change the job manager port to make this work?

Any suggestions would be highly appreciated.

Thanks.

Patrick

[Patrick Eifler]

Hi Chesney,

Thanks for the hint. 

I have mounted my certs in both job and taskmanager volume mounts. 

When the containers bootup I get the log that the ssl store is successfully loaded.

Note: I use the same keystore setup to connect to secured Kafka Cluster and this works.

How would you suggest the ssl setup to be done?

Thanks 

Chesnay Schepler <[hidden email]> schrieb am Fr. 6. Nov. 2020 um 00:03:

It is unlikely that this is a port issue, and I would currently suspect that something in your SSL setup is not correct.

@Nico: do you have a suggestion on how to debug this?

On 11/5/2020 4:23 PM, Patrick Eifler wrote:

Hi,

I did set up a flink session cluster on K8s. 

Now I added the ssl configuration as shown in the documentation:

# Flink TLS

security.ssl.internal.enabled: true

security.ssl.internal.keystore: /config/internal-keystore/internal.keystore.jks

security.ssl.internal.truststore: /config/internal-keystore/internal.keystore.jks

security.ssl.internal.keystore-password: {{ .Values.keystore.password }}

security.ssl.internal.truststore-password: {{ .Values.keystore.password }}

security.ssl.internal.key-password: {{ .Values.keystore.password }}

Now I get the problem that the task manager cannot connect to the job manager nor the resource manager:

could not resolve ResourceManager address akka.ssl.tcp://flink@flink-sc-jobmanager:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.ssl.tcp://flink@flink-sc-jobmanager:6123/user/rpc/resourcemanager_*.

Do I need to change the job manager port to make this work?

Any suggestions would be highly appreciated.

Thanks.

Patrick