StreamingFileSink: any risk parallelizing active buckets checkpointing?

[Paul Bernier]

Hi experts,

 

I am trying to use S3 StreamingFileSink with a high number of active buckets (>1000). I found that checkpointing duration will grow linearly with the number of active buckets, which makes achieving high number of active buckets difficult. One reason for that is the each active buckets are snapshotted sequentially in a loop. Given that operation involves waiting for some data to finish being uploaded to S3 that can become quite a long wait.

 

My question is: could this loop be safely multi-threaded?

Each Bucket seems independent (they do share the bucketWriter though). I have also done some basic prototyping and validation and it looks ok. So I wondering if I am overlooking anything and if my approach is viable?

 

Note: the same approach would also need to be applied to the onSuccessfulCompletionOfCheckpoint step with this while loop committing files to S3.

 

Thank you.

 

Paul

[Till Rohrmann]

I've responded on the dev ML. Let's continue the discussion there: https://lists.apache.org/thread.html/r251e395c759193d9c75f97b8bfc4917772219ea48bb1848ccc23d26e%40%3Cdev.flink.apache.org%3E

Cheers,

Till

On Thu, Jul 30, 2020 at 8:57 PM Paul Bernier <[hidden email]> wrote:

Hi experts,

 

I am trying to use S3 StreamingFileSink with a high number of active buckets (>1000). I found that checkpointing duration will grow linearly with the number of active buckets, which makes achieving high number of active buckets difficult. One reason for that is the each active buckets are snapshotted sequentially in a loop. Given that operation involves waiting for some data to finish being uploaded to S3 that can become quite a long wait.

 

My question is: could this loop be safely multi-threaded?

Each Bucket seems independent (they do share the bucketWriter though). I have also done some basic prototyping and validation and it looks ok. So I wondering if I am overlooking anything and if my approach is viable?

 

Note: the same approach would also need to be applied to the onSuccessfulCompletionOfCheckpoint step with this while loop committing files to S3.

 

Thank you.

 

Paul