Recommendation for BucketingSink to GZIP compress files to S3

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Recommendation for BucketingSink to GZIP compress files to S3

Flink Developer
Hi, what is the recommended method for using BucketingSink and compressing files using GZIP before it is uploaded to S3?

I read that one way is to extend the StreamWriterBase class and wrap the stream using GZIPOutputStream. Is there an Flink example for this? If so, what would be the proper way to override the methods (open(), write(), close(), flush())?

Thank you.