Re: How does BucketingSink generate a SUCCESS file when a directory is finished

Posted by Ben Yan on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/How-does-BucketingSink-generate-a-SUCCESS-file-when-a-directory-is-finished-tp18072p18077.html

Hi:

I think so too! But I have a question that when should I add this logic in BucketingSink! And who does this logic, and ensures that the logic is executed only once, not every parallel instance of the sink that executes this logic!

Best,
Ben

> On 31 Jan 2018, at 5:58 PM, Hung <[hidden email]> wrote:
>
> it depends on how you partition your file. in my case I write file per hour,
> so I'm sure that file is ready after that hour period, in processing time.
> Here, read to be ready means this file contains all the data in that hour
> period.
>
> If the downstream runs in a batch way, you may want to ensure the file is
> ready.
> In this case, ready to read can mean all the data before watermark as
> arrived.
> You could take the BucketingSink and implement this logic there, maybe wait
> until watermark
> reaches
>
> Best,
>
> Sendoh
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/