(DEPRECATED) Apache Flink User Mailing List archive.

Flink S3 error

classic Classic list List threaded Threaded
♦
♦
 2 messages Options
Harrison Xu
Reply | Threaded
Open this post in threaded view
♦
♦
|

Flink S3 error

Harrison Xu
I'm seeing this exception with the S3 uploader - it claims a previously part file was not found. Full jobmanager logs attached. (Flink 1.8)

java.io.FileNotFoundException: No such file or directory: s3a://qcache/tmp/kafka/meta/rq_features/dt=2019-10-30T15/partition_1/_part-4-1169_tmp_21400e5e-3921-4f33-a980-ac953b50b4b7
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2255)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:699)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.FileSystem.open(FileSystem.java:950)
	at org.apache.flink.fs.s3hadoop.HadoopS3AccessHelper.getObject(HadoopS3AccessHelper.java:98)
	at org.apache.flink.fs.s3.common.writer.S3RecoverableMultipartUploadFactory.recoverInProgressPart(S3RecoverableMultipartUploadFactory.java:97)
	at org.apache.flink.fs.s3.common.writer.S3RecoverableMultipartUploadFactory.recoverRecoverableUpload(S3RecoverableMultipartUploadFactory.java:75)
	at org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.recover(S3RecoverableWriter.java:95)
	at org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.recover(S3RecoverableWriter.java:50)
	at com.quora.dataInfra.s3connector.flink.filesystem.Bucket.restoreInProgressFile(Bucket.java:146)
	at com.quora.dataInfra.s3connector.flink.filesystem.Bucket.<init>(Bucket.java:133)
	at com.quora.dataInfra.s3connector.flink.filesystem.Bucket.restore(Bucket.java:404)
	at com.quora.dataInfra.s3connector.flink.filesystem.DefaultBucketFactoryImpl.restoreBucket(DefaultBucketFactoryImpl.java:67)
	at com.quora.dataInfra.s3connector.flink.filesystem.Buckets.handleRestoredBucketState(Buckets.java:182)
	at com.quora.dataInfra.s3connector.flink.filesystem.Buckets.initializeActiveBuckets(Buckets.java:170)
	at com.quora.dataInfra.s3connector.flink.filesystem.Buckets.initializeState(Buckets.java:154)
	at com.quora.dataInfra.s3connector.flink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:344)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
	at java.lang.Thread.run(Thread.java:748)
vino yang
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Flink S3 error

vino yang
Hi Harrison,

So did you check whether the file exists or not? And what's your question?

Best,
Vino

Harrison Xu <[hidden email]> 于2019年10月31日周四 上午5:24写道：
I'm seeing this exception with the S3 uploader - it claims a previously part file was not found. Full jobmanager logs attached. (Flink 1.8)

java.io.FileNotFoundException: No such file or directory: s3a://qcache/tmp/kafka/meta/rq_features/dt=2019-10-30T15/partition_1/_part-4-1169_tmp_21400e5e-3921-4f33-a980-ac953b50b4b7
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2255)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:699)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.FileSystem.open(FileSystem.java:950)
	at org.apache.flink.fs.s3hadoop.HadoopS3AccessHelper.getObject(HadoopS3AccessHelper.java:98)
	at org.apache.flink.fs.s3.common.writer.S3RecoverableMultipartUploadFactory.recoverInProgressPart(S3RecoverableMultipartUploadFactory.java:97)
	at org.apache.flink.fs.s3.common.writer.S3RecoverableMultipartUploadFactory.recoverRecoverableUpload(S3RecoverableMultipartUploadFactory.java:75)
	at org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.recover(S3RecoverableWriter.java:95)
	at org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.recover(S3RecoverableWriter.java:50)
	at com.quora.dataInfra.s3connector.flink.filesystem.Bucket.restoreInProgressFile(Bucket.java:146)
	at com.quora.dataInfra.s3connector.flink.filesystem.Bucket.<init>(Bucket.java:133)
	at com.quora.dataInfra.s3connector.flink.filesystem.Bucket.restore(Bucket.java:404)
	at com.quora.dataInfra.s3connector.flink.filesystem.DefaultBucketFactoryImpl.restoreBucket(DefaultBucketFactoryImpl.java:67)
	at com.quora.dataInfra.s3connector.flink.filesystem.Buckets.handleRestoredBucketState(Buckets.java:182)
	at com.quora.dataInfra.s3connector.flink.filesystem.Buckets.initializeActiveBuckets(Buckets.java:170)
	at com.quora.dataInfra.s3connector.flink.filesystem.Buckets.initializeState(Buckets.java:154)
	at com.quora.dataInfra.s3connector.flink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:344)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
	at java.lang.Thread.run(Thread.java:748)
Free forum by Nabble Edit this page