Flink 1.5, failed to instantiate S3 FS

[Hao Sun]

I can not find anywhere I have 100M. Not sure why I get this failure.

This is in my dev docker env. Same configure file worked well for 1.3.2

===== Log ====

Caused by: org.apache.flink.util.FlinkException: Failed to submit job aa75905062dd0487034bb9d8b6617dc2.

        at org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:254)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)

        at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)

        at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70)

        at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)

        at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)

        at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)

        at akka.actor.Actor$class.aroundReceive(Actor.scala:502)

        at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)

        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)

        at akka.actor.ActorCell.invoke(ActorCell.scala:495)

        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)

        at akka.dispatch.Mailbox.run(Mailbox.scala:224)

        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)

        ... 4 more

Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager

        at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:169)

        at org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:885)

        at org.apache.flink.runtime.dispatcher.Dispatcher.createJobManagerRunner(Dispatcher.java:287)

        at org.apache.flink.runtime.dispatcher.Dispatcher.runJob(Dispatcher.java:277)

        at org.apache.flink.runtime.dispatcher.Dispatcher.persistAndRunJob(Dispatcher.java:262)

        at org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:249)

        ... 21 more

Caused by: java.lang.RuntimeException: Failed to start checkpoint ID counter: Cannot instantiate file system for URI: s3a://zendesk-dev-orca-fps/pod0/checkpoints-meta

        at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:253)

        at org.apache.flink.runtime.executiongraph.ExecutionGraph.enableCheckpointing(ExecutionGraph.java:495)

        at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:345)

        at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)

        at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1150)

        at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1130)

        at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:298)

        at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:151)

        ... 26 more

Caused by: java.io.IOException: Cannot instantiate file system for URI: s3a://zendesk-dev-orca-fps/pod0/checkpoints-meta

        at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:187)

        at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)

        at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)

        at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)

        at org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:61)

        at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:441)

        at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createCheckpointStorage(RocksDBStateBackend.java:379)

        at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:247)

        ... 33 more

Caused by: java.lang.NumberFormatException: For input string: "100M"

        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

        at java.lang.Long.parseLong(Long.java:589)

        at java.lang.Long.parseLong(Long.java:631)

        at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1429)

        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:248)

        at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:159)

        ... 40 more

[Amit Jain]

Hi Hao,

Have look over https://issues.apache.org/jira/browse/HADOOP-13811?focusedCommentId=15703276&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15703276

What version of Hadoop are you using? Could you provide classpath used
by Flink Job Manager, it is present in jobmanager.log file.

--
Cheers,
Amit

[Hao Sun]

Thanks Amit for checking. I do not use hadoop, but I am using Flink with bundled HDP 2.8 binary. I think this article is right, I mixed 2.7 lib and 2.8 binary somehow.

On Sat, Jun 2, 2018 at 1:05 AM Amit Jain <[hidden email]> wrote:

Hi Hao,

Have look over https://issues.apache.org/jira/browse/HADOOP-13811?focusedCommentId=15703276&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15703276

What version of Hadoop are you using? Could you provide classpath used
by Flink Job Manager, it is present in jobmanager.log file.

--
Cheers,
Amit