Flink 1.10 on MapR secure cluster with high availability

[Maxim Parkachov]

Hi everyone,

I have already written about issue with Flink 1.9 on secure MapR cluster and high availability. The issue was resolved with custom compiled Flink with vendor mapr repositories enabled. The history could be found https://www.mail-archive.com/user@.../msg28235.html

Unfortunately, in current 1.10 RC vendor repositories were removed and I'm failing to get working configuration.  Current situation with 1.10 RC and secure MapR cluster:

1. Without HA, Flink uses class path provided zookeeper jar (mapr specific) and everything works fine.

2. With HA enabled, Flink uses shaded zookeeper  (org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn) which  doesn't have MapR specific changes and fails to authenticate.

I would really appreciate any help in resolving this issue.I'm ready to provide any required details.

Regards,

Maxim.

[Chesnay Schepler]

You must rebuild Flink while overriding zookeeper.version property to match your MapR setup.

For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604

Note that you will also have to configure the MapR repository in your local setup as described here.

On 05/02/2020 15:12, Maxim Parkachov wrote:

Hi everyone,

I have already written about issue with Flink 1.9 on secure MapR cluster and high availability. The issue was resolved with custom compiled Flink with vendor mapr repositories enabled. The history could be found https://www.mail-archive.com/user@.../msg28235.html

Unfortunately, in current 1.10 RC vendor repositories were removed and I'm failing to get working configuration.  Current situation with 1.10 RC and secure MapR cluster:

1. Without HA, Flink uses class path provided zookeeper jar (mapr specific) and everything works fine.

2. With HA enabled, Flink uses shaded zookeeper  (org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn) which  doesn't have MapR specific changes and fails to authenticate.

I would really appreciate any help in resolving this issue.I'm ready to provide any required details.

Regards,

Maxim.

[Maxim Parkachov]

Hi Chesnay,

thanks for advise. Will it work if I include MapR specific zookeeper in job dependencies and still use out-of-box Flink binary distribution ?

Regards,

Maxim.

On Wed, Feb 5, 2020 at 3:25 PM Chesnay Schepler <[hidden email]> wrote:

You must rebuild Flink while overriding zookeeper.version property to match your MapR setup.

For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604

Note that you will also have to configure the MapR repository in your local setup as described here.

On 05/02/2020 15:12, Maxim Parkachov wrote:

Hi everyone,

I have already written about issue with Flink 1.9 on secure MapR cluster and high availability. The issue was resolved with custom compiled Flink with vendor mapr repositories enabled. The history could be found https://www.mail-archive.com/user@.../msg28235.html

Unfortunately, in current 1.10 RC vendor repositories were removed and I'm failing to get working configuration.  Current situation with 1.10 RC and secure MapR cluster:

1. Without HA, Flink uses class path provided zookeeper jar (mapr specific) and everything works fine.

2. With HA enabled, Flink uses shaded zookeeper  (org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn) which  doesn't have MapR specific changes and fails to authenticate.

I would really appreciate any help in resolving this issue.I'm ready to provide any required details.

Regards,

Maxim.

[Chesnay Schepler]

No, since a) HA will never use classes from the user-jar and b) zookeeper is relocated to a different package (to avoid conflicts) and hence any replacement has to follow the same relocation convention.

On 05/02/2020 15:38, Maxim Parkachov wrote:

Hi Chesnay,

thanks for advise. Will it work if I include MapR specific zookeeper in job dependencies and still use out-of-box Flink binary distribution ?

Regards,

Maxim.

On Wed, Feb 5, 2020 at 3:25 PM Chesnay Schepler <[hidden email]> wrote:

You must rebuild Flink while overriding zookeeper.version property to match your MapR setup.

For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604

Note that you will also have to configure the MapR repository in your local setup as described here.

On 05/02/2020 15:12, Maxim Parkachov wrote:

Hi everyone,

I have already written about issue with Flink 1.9 on secure MapR cluster and high availability. The issue was resolved with custom compiled Flink with vendor mapr repositories enabled. The history could be found https://www.mail-archive.com/user@.../msg28235.html

Unfortunately, in current 1.10 RC vendor repositories were removed and I'm failing to get working configuration.  Current situation with 1.10 RC and secure MapR cluster:

1. Without HA, Flink uses class path provided zookeeper jar (mapr specific) and everything works fine.

2. With HA enabled, Flink uses shaded zookeeper  (org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn) which  doesn't have MapR specific changes and fails to authenticate.

I would really appreciate any help in resolving this issue.I'm ready to provide any required details.

Regards,

Maxim.

[Maxim Parkachov]

Hi Chesnay,

I managed to re-compile with MapR zookeeper and can confirm that it works with HA as well.

Still I find it strange that HA uses shadow version of zookeeper instead of version from classpath how it is done for hadoop.

Thanks,

Maxim.

On Wed, Feb 5, 2020 at 3:43 PM Chesnay Schepler <[hidden email]> wrote:

No, since a) HA will never use classes from the user-jar and b) zookeeper is relocated to a different package (to avoid conflicts) and hence any replacement has to follow the same relocation convention.

On 05/02/2020 15:38, Maxim Parkachov wrote:

Hi Chesnay,

thanks for advise. Will it work if I include MapR specific zookeeper in job dependencies and still use out-of-box Flink binary distribution ?

Regards,

Maxim.

On Wed, Feb 5, 2020 at 3:25 PM Chesnay Schepler <[hidden email]> wrote:

You must rebuild Flink while overriding zookeeper.version property to match your MapR setup.

For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604

Note that you will also have to configure the MapR repository in your local setup as described here.

On 05/02/2020 15:12, Maxim Parkachov wrote:

Hi everyone,

I have already written about issue with Flink 1.9 on secure MapR cluster and high availability. The issue was resolved with custom compiled Flink with vendor mapr repositories enabled. The history could be found https://www.mail-archive.com/user@.../msg28235.html

Unfortunately, in current 1.10 RC vendor repositories were removed and I'm failing to get working configuration.  Current situation with 1.10 RC and secure MapR cluster:

1. Without HA, Flink uses class path provided zookeeper jar (mapr specific) and everything works fine.

2. With HA enabled, Flink uses shaded zookeeper  (org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn) which  doesn't have MapR specific changes and fails to authenticate.

I would really appreciate any help in resolving this issue.I'm ready to provide any required details.

Regards,

Maxim.

[Arvid Heise-3]

Hi Maxim,

in general, we have used shading in the past to avoid dependency hells for users. We are in the process to replace more and more relocations with plugins having a different classloader. However, I can't tell you if that will also work for zookeeper and when that would happen.

If you manage to find out how Hadoop is solving that, I'd appreciate if you could let us know.

Best,

Arvid

On Fri, Feb 7, 2020 at 5:02 PM Maxim Parkachov <[hidden email]> wrote:

Hi Chesnay,

I managed to re-compile with MapR zookeeper and can confirm that it works with HA as well.

Still I find it strange that HA uses shadow version of zookeeper instead of version from classpath how it is done for hadoop.

Thanks,

Maxim.

On Wed, Feb 5, 2020 at 3:43 PM Chesnay Schepler <[hidden email]> wrote:

No, since a) HA will never use classes from the user-jar and b) zookeeper is relocated to a different package (to avoid conflicts) and hence any replacement has to follow the same relocation convention.

On 05/02/2020 15:38, Maxim Parkachov wrote:

Hi Chesnay,

thanks for advise. Will it work if I include MapR specific zookeeper in job dependencies and still use out-of-box Flink binary distribution ?

Regards,

Maxim.

On Wed, Feb 5, 2020 at 3:25 PM Chesnay Schepler <[hidden email]> wrote:

You must rebuild Flink while overriding zookeeper.version property to match your MapR setup.

For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604

Note that you will also have to configure the MapR repository in your local setup as described here.

On 05/02/2020 15:12, Maxim Parkachov wrote:

Hi everyone,

I have already written about issue with Flink 1.9 on secure MapR cluster and high availability. The issue was resolved with custom compiled Flink with vendor mapr repositories enabled. The history could be found https://www.mail-archive.com/user@.../msg28235.html

Unfortunately, in current 1.10 RC vendor repositories were removed and I'm failing to get working configuration.  Current situation with 1.10 RC and secure MapR cluster:

1. Without HA, Flink uses class path provided zookeeper jar (mapr specific) and everything works fine.

2. With HA enabled, Flink uses shaded zookeeper  (org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn) which  doesn't have MapR specific changes and fails to authenticate.

I would really appreciate any help in resolving this issue.I'm ready to provide any required details.

Regards,

Maxim.