Bucketing HDFS Sink Failing randomly after fews days it runs successfully

[Raja.Aravapalli]

 

Hi Team,

 

I have a Bucketing Sink writing to HDFS files…. Which is running successfully for 4days failing suddenly with below exception:

 

Caused by: java.io.IOException: Cannot find required BLOB at /tmp/blobStore

 

 

Code below:

 

BucketingSink<String> HdfsSink = new BucketingSink<String> (hdfsOutputPath);

HdfsSink.setBucketer(new BasePathBucketer<String>());
HdfsSink.setBatchSize(1024 * 1024 * hdfsOutputBatchSizeInMB); // this means 'hdfsOutputBatchSizeInMB' MB
HdfsSink.setPartPrefix("PART-FILE-" + Long.toString(System.currentTimeMillis()));

 

 

Can someone please share thoughts on how I  can fix this.

 

Thanks.

 

 

Regards,

Raja.