Will StreamingFileSink.forBulkFormat(...) support overriding OnCheckpointRollingPolicy?

[Elkhan Dadashov]

Hi folks,

Will StreamingFileSink.forBulkFormat(...) support overriding OnCheckpointRollingPolicy?

Does anyone use StreamingFileSink with checkpoint disabled for writing Parquet output files?

The output parquet files are generated, but they are empty, and stay in inprogress state, even when the job completes:

.part-0-0.inprogress.3e31ba42-588c-48cc-ad6d-d0ebcf1d8632

.part-1-0.inprogress.78e1f1dc-3c1c-417b-8270-2bf0298f985a

.part-2-0.inprogress.087cf3f1-7e2d-4a03-a518-62f576ed7eea

Exactly-once semantics is not important for my case, would then using BucketingSink the only option ?

Thanks.