Flink S3 Hadoop dependencies

[Satish Saley]

Hi team,

Was there a reason for not shading hadoop-common https://github.com/apache/flink/commit/e1e7d7f7ecc080c850a264021bf1b20e3d27d373#diff-e7b798a682ee84ab804988165e99761cR38-R44 ? This is leaking lots of classes such as guava and causing issues in our flink application.
I see that hadoop-common classes were shaded in earlier versions https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-hadoop/1.9.0

Stacktrace :

Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V
at io.grpc.Metadata$Key.validateName(Metadata.java:742)
at io.grpc.Metadata$Key.<init>(Metadata.java:750)
at io.grpc.Metadata$Key.<init>(Metadata.java:668)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
at io.grpc.Metadata$Key.of(Metadata.java:705)
at io.grpc.Metadata$Key.of(Metadata.java:701)
at io.grpc.internal.GrpcUtil.<clinit>(GrpcUtil.java:80)
at io.grpc.internal.AbstractManagedChannelImplBuilder.<clinit>(AbstractManagedChannelImplBuilder.java:90)
at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:42)
at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:23)
at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:76)

[Chesnay Schepler]

Filesystems are supposed to be used as plugins (by putting the jars under plugins/ instead of lib/), in which case they are loaded separately from other classes, specifically user-code.

https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/plugins.html

On 14/08/2020 20:25, Satish Saley wrote:

Hi team,

Was there a reason for not shading hadoop-common https://github.com/apache/flink/commit/e1e7d7f7ecc080c850a264021bf1b20e3d27d373#diff-e7b798a682ee84ab804988165e99761cR38-R44 ? This is leaking lots of classes such as guava and causing issues in our flink application.
I see that hadoop-common classes were shaded in earlier versions https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-hadoop/1.9.0

Stacktrace :

Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V
at io.grpc.Metadata$Key.validateName(Metadata.java:742)
at io.grpc.Metadata$Key.<init>(Metadata.java:750)
at io.grpc.Metadata$Key.<init>(Metadata.java:668)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
at io.grpc.Metadata$Key.of(Metadata.java:705)
at io.grpc.Metadata$Key.of(Metadata.java:701)
at io.grpc.internal.GrpcUtil.<clinit>(GrpcUtil.java:80)
at io.grpc.internal.AbstractManagedChannelImplBuilder.<clinit>(AbstractManagedChannelImplBuilder.java:90)
at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:42)
at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:23)
at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:76)