BucketingSink compressed text exactlyonce

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

BucketingSink compressed text exactlyonce

ysn2233
Hi guys
Any good ideas to achieve exactly once BucketingSink for text file?truncating compressed binary file will corrupt the gzip file which means I need to -text that gzip and redirect to a text file then compressed it agan and finally upload to hdfs. Its really inefficient. Any other compressed format suitable for flink‘s fault tolerence mechanism?Ty

Sent from my iPhone