Flink jobmanager TLS connectivity to Zookeeper

[Azeem Mufti]

 I'm trying to figure out a way to make Flink jobmanager (in HA) connect to zookeeper over SSL/TLS. It doesn't seem like there are native properties like Kafka has that support this interaction yet. Is this true or is there some way that I can go about doing this?

[Matthias]

Hi Azeem,

I haven't worked with Flink's SSL support, yet. But have you taken a look at the SSL configuration options listed under [1]?

Best,

Matthias

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/security/security-ssl.html#complete-list-of-ssl-options

On Tue, Dec 8, 2020 at 8:01 PM Azeem Mufti <[hidden email]> wrote:

 I'm trying to figure out a way to make Flink jobmanager (in HA) connect to zookeeper over SSL/TLS. It doesn't seem like there are native properties like Kafka has that support this interaction yet. Is this true or is there some way that I can go about doing this?

--

[Azeem Mufti]

Hey Matthias,

I have and it doesn't seem like there are any native properties that support this interaction. I did try enabling the rest/internal SSL properties to see if that would work but when my jobmanager tries to make a connection to zookeeper, zookeeper is rejecting the connection saying it's not a TLS/SSL record.

Thanks,

Azeem 

On Thu, Dec 10, 2020 at 9:36 AM Matthias Pohl <[hidden email]> wrote:

Hi Azeem,

I haven't worked with Flink's SSL support, yet. But have you taken a look at the SSL configuration options listed under [1]?

Best,

Matthias

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/security/security-ssl.html#complete-list-of-ssl-options

On Tue, Dec 8, 2020 at 8:01 PM Azeem Mufti <[hidden email]> wrote:

 I'm trying to figure out a way to make Flink jobmanager (in HA) connect to zookeeper over SSL/TLS. It doesn't seem like there are native properties like Kafka has that support this interaction yet. Is this true or is there some way that I can go about doing this?

--

Matthias Pohl | Engineer

Follow us @VervericaData Ververica

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton Wehner

[rmetzger0]

Hey Azeem,

I haven't tried this myself, but from the code / documentation, this could work:

Flink ships with ZK 3.4 by default. You need to remove the ZK3.4 jar file from the lib/ folder and add the ZK3.5 file from opt/ to lib/.

According to this guide, you could try passing the SSL configuration values via JVM properties to the ZK client: https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide

Setting the Flink config "env.java.opts" to -Dzookeeper.clientCnxnSocket="org.apache.zookeeper.ClientCnxnSocketNetty" -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location="/path/to/your/keystore" and so on ... 

You might be the first human on this planet trying to configure SSL for ZK in Flink, so things might not work out of the box. The Flink logs on DEBUG level will probably be helpful to ensure that ZK picks up the configuration keys.

If you figure out how to get it working, a contribution to the Flink docs will be much appreciated ;)

On Mon, Dec 14, 2020 at 5:47 PM Azeem Mufti <[hidden email]> wrote:

Hey Matthias,

I have and it doesn't seem like there are any native properties that support this interaction. I did try enabling the rest/internal SSL properties to see if that would work but when my jobmanager tries to make a connection to zookeeper, zookeeper is rejecting the connection saying it's not a TLS/SSL record.

Thanks,

Azeem 

On Thu, Dec 10, 2020 at 9:36 AM Matthias Pohl <[hidden email]> wrote:

Hi Azeem,

I haven't worked with Flink's SSL support, yet. But have you taken a look at the SSL configuration options listed under [1]?

Best,

Matthias

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/security/security-ssl.html#complete-list-of-ssl-options

On Tue, Dec 8, 2020 at 8:01 PM Azeem Mufti <[hidden email]> wrote:

 I'm trying to figure out a way to make Flink jobmanager (in HA) connect to zookeeper over SSL/TLS. It doesn't seem like there are native properties like Kafka has that support this interaction yet. Is this true or is there some way that I can go about doing this?

--

Matthias Pohl | Engineer

Follow us @VervericaData Ververica

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton Wehner