NoSuchMethodError: org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.<init>(Ljava/lang/String;Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader

[Claude Murad]

Hello, 

I'm trying to upgrade Flink from 1.7 to 1.10 retaining our Hadoop integration.  I copied the jar file flink-shaded-hadoop-2-uber-2.7.5-10.0.jar into /opt/flink/lib.  I also copied the files flink-s3-fs-hadoop-1.10.0.jar and flink-s3-fs-presto-1.10.0.jar into /opt/flink/plugins/s3 folder.  The error below occurs after deploying and launching docker image 1.10.0-scala_2.11.  I saw that S3FileSystemFactory.java is now importing org.apache.flink.runtime.util.HadoopConfigLoader instead of org.apache.flink.fs.s3.common.HadoopConfigLoader which is how it was before.  I see the jar file flink-dist_2.11-1.10.0.jar contains the org.apache.flink.runtime.util.HadoopConfigLoader and it is under the folder /opt/flink/lib.  Any ideas on how to resolve this error?  Any help would be greatly appreciated, thank you. 

ERROR org.apache.flink.core.fs.FileSystem                           - Failed to load a file system via services
java.util.ServiceConfigurationError: org.apache.flink.core.fs.FileSystemFactory: Provider org.apache.flink.fs.s3presto.S3PFileSystemFactory could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1024)
at org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1006)
at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:303)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.configureFileSystems(ClusterEntrypoint.java:194)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:164)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:518)
at org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:64)
Caused by: java.lang.NoSuchMethodError: org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.<init>(Ljava/lang/String;Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader;)V
at org.apache.flink.fs.s3presto.S3FileSystemFactory.<init>(S3FileSystemFactory.java:50)
at org.apache.flink.fs.s3presto.S3PFileSystemFactory.<init>(S3PFileSystemFactory.java:24)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)

[Guowei Ma]

Hi,

In 1.10 there is no 'Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader' . So I think there might be a legacy S3FileSystemFactory in your jar. You could check whether there is a 'org.apache.flink.fs.s3presto.common.HadoopConfigLoader' in your jar or not. If there is one you could remove the old S3FileSystemFactory and try again.

Btw I think you might not copy both flink-s3-fs-hadoop-1.10.0 and  flink-s3-fs-presto-1.10.0.jar to the same plugin dir. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins

Claude Murad <[hidden email]> 于2020年6月10日周三 上午4:06写道：

Hello, 

I'm trying to upgrade Flink from 1.7 to 1.10 retaining our Hadoop integration.  I copied the jar file flink-shaded-hadoop-2-uber-2.7.5-10.0.jar into /opt/flink/lib.  I also copied the files flink-s3-fs-hadoop-1.10.0.jar and flink-s3-fs-presto-1.10.0.jar into /opt/flink/plugins/s3 folder.  The error below occurs after deploying and launching docker image 1.10.0-scala_2.11.  I saw that S3FileSystemFactory.java is now importing org.apache.flink.runtime.util.HadoopConfigLoader instead of org.apache.flink.fs.s3.common.HadoopConfigLoader which is how it was before.  I see the jar file flink-dist_2.11-1.10.0.jar contains the org.apache.flink.runtime.util.HadoopConfigLoader and it is under the folder /opt/flink/lib.  Any ideas on how to resolve this error?  Any help would be greatly appreciated, thank you. 

ERROR org.apache.flink.core.fs.FileSystem                           - Failed to load a file system via services
java.util.ServiceConfigurationError: org.apache.flink.core.fs.FileSystemFactory: Provider org.apache.flink.fs.s3presto.S3PFileSystemFactory could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1024)
at org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1006)
at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:303)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.configureFileSystems(ClusterEntrypoint.java:194)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:164)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:518)
at org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:64)
Caused by: java.lang.NoSuchMethodError: org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.<init>(Ljava/lang/String;Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader;)V
at org.apache.flink.fs.s3presto.S3FileSystemFactory.<init>(S3FileSystemFactory.java:50)
at org.apache.flink.fs.s3presto.S3PFileSystemFactory.<init>(S3PFileSystemFactory.java:24)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)