StreamingFileSink with PrestoFS?

[Jared Stehler]

I'm encountering an error on init with StreamingFileSink and presto-fs; before I continue down what appears to be a classpath issue, can someone stop me if StreamingFileSink doesn't support presto-fs?

Error I'm seeing is: 

java.lang.UnsupportedOperationException: Not implemented by the PrestoS3FileSystem FileSystem implementation

        at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:219)

        at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter.<init>(HadoopRecoverableWriter.java:56)

        at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)

        at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)

        at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:111)

        at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$BulkFormatBuilder.createBuckets(StreamingFileSink.java:317)

        at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327)

        at com.intellify.flink.crusher.executor.sink.TracingSourceRecordSink.initializeState(TracingSourceRecordSink.java:105)

        at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)

        at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)

        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)

        at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:254)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)

[Aljoscha Krettek]

Hi Jared,

using the new StreamingFileSink is indeed not supported for S3 (PrestoFS or not) right now. Work on this is tracked under https://issues.apache.org/jira/browse/FLINK-9752 and should hopefully make it into the next Flink release.

Best,

Aljoscha

On 13. Sep 2018, at 14:40, Jared Stehler <[hidden email]> wrote:

I'm encountering an error on init with StreamingFileSink and presto-fs; before I continue down what appears to be a classpath issue, can someone stop me if StreamingFileSink doesn't support presto-fs?

Error I'm seeing is: 

java.lang.UnsupportedOperationException: Not implemented by the PrestoS3FileSystem FileSystem implementation

        at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:219)

        at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter.<init>(HadoopRecoverableWriter.java:56)

        at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)

        at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)

        at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:111)

        at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$BulkFormatBuilder.createBuckets(StreamingFileSink.java:317)

        at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327)

        at com.intellify.flink.crusher.executor.sink.TracingSourceRecordSink.initializeState(TracingSourceRecordSink.java:105)

        at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)

        at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)

        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)

        at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:254)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)

[Jared Stehler]

Aha, thanks!

On Thu, Sep 13, 2018 at 2:53 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi Jared,

using the new StreamingFileSink is indeed not supported for S3 (PrestoFS or not) right now. Work on this is tracked under https://issues.apache.org/jira/browse/FLINK-9752 and should hopefully make it into the next Flink release.

Best,

Aljoscha

On 13. Sep 2018, at 14:40, Jared Stehler <[hidden email]> wrote:

I'm encountering an error on init with StreamingFileSink and presto-fs; before I continue down what appears to be a classpath issue, can someone stop me if StreamingFileSink doesn't support presto-fs?

Error I'm seeing is: 

java.lang.UnsupportedOperationException: Not implemented by the PrestoS3FileSystem FileSystem implementation

        at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:219)

        at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter.<init>(HadoopRecoverableWriter.java:56)

        at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)

        at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)

        at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:111)

        at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$BulkFormatBuilder.createBuckets(StreamingFileSink.java:317)

        at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327)

        at com.intellify.flink.crusher.executor.sink.TracingSourceRecordSink.initializeState(TracingSourceRecordSink.java:105)

        at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)

        at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)

        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)

        at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:254)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)