Looking for example for bucketingSink / StreamingFileSink

[avilevi]

Hi Guys,

very new to flink so my apology for the newbie questions :)

but I desperately looking for a good example for streaming to file using bucketingSink / StreamingFileSink . Unfortunately the examples in the documentation are not event compiling (at least not the ones in scala https://issues.apache.org/jira/browse/FLINK-11053 )

I tried using bucketing sink with streamingFileSink (or just streamingFileSink ) and finally tried to implement a writer but with no luck. 

BucketingSink seems to be a perfect fit because I can set the batch interval by time interval or size which is exactly what I need.

This is my last attempt (sample project) which results a lot of "pending" files. 

Any help would be appreciated

Thanks

Avi

[miki haiat]

HI Avi , 
Im assuming that the cause  of the "pending" file is because the checkpoint isn't finished successfully [1]

Can you try to change the checkpoint time to 1 min as well .

Thanks,

Miki

[1] https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L131  

On Mon, Dec 3, 2018 at 12:45 PM Avi Levi <[hidden email]> wrote:

Hi Guys,

very new to flink so my apology for the newbie questions :)

but I desperately looking for a good example for streaming to file using bucketingSink / StreamingFileSink . Unfortunately the examples in the documentation are not event compiling (at least not the ones in scala https://issues.apache.org/jira/browse/FLINK-11053 )

I tried using bucketing sink with streamingFileSink (or just streamingFileSink ) and finally tried to implement a writer but with no luck. 

BucketingSink seems to be a perfect fit because I can set the batch interval by time interval or size which is exactly what I need.

This is my last attempt (sample project) which results a lot of "pending" files. 

Any help would be appreciated

Thanks

Avi