I'm encountering an error on init with StreamingFileSink and presto-fs; before I continue down what appears to be a classpath issue, can someone stop me if StreamingFileSink doesn't support presto-fs? Error I'm seeing is: java.lang.UnsupportedOperationException: Not implemented by the PrestoS3FileSystem FileSystem implementation at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:219) at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter.<init>(HadoopRecoverableWriter.java:56) at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202) at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69) at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:111) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$BulkFormatBuilder.createBuckets(StreamingFileSink.java:317) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327) at com.intellify.flink.crusher.executor.sink.TracingSourceRecordSink.initializeState(TracingSourceRecordSink.java:105) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:254) at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) |
Hi Jared,
using the new StreamingFileSink is indeed not supported for S3 (PrestoFS or not) right now. Work on this is tracked under https://issues.apache.org/jira/browse/FLINK-9752 and should hopefully make it into the next Flink release. Best, Aljoscha
|
Aha, thanks! On Thu, Sep 13, 2018 at 2:53 PM, Aljoscha Krettek <[hidden email]> wrote:
|
Free forum by Nabble | Edit this page |