Server side error: Cannot find required BLOB at /tmp/blobStore

[Raja.Aravapalli]

Hi,

 

My Flink application which reads from Kafka and writes to HDFS is failing repeatedly with below exception:

 

Caused by: java.io.IOException: Server side error: Cannot find required BLOB at /tmp/blobStore-**********

 

Can someone please help me on, what could be the root cause of this issue?  I am not able to trace logs also.

 

Also, FYI, our FLINK cluster(4nodes) is setup on our Hadoop YARN cluster.

 

Thanks a lot.

 

 

Regards,

Raja.

[miki haiat]

/tmp/blobStore 

Is it the path  for checkpoints/savepoints  storage ?

On Tue, 11 Sep 2018, 10:01 Raja.Aravapalli, <[hidden email]> wrote:

Hi,

 

My Flink application which reads from Kafka and writes to HDFS is failing repeatedly with below exception:

 

Caused by: java.io.IOException: Server side error: Cannot find required BLOB at /tmp/blobStore-**********

 

Can someone please help me on, what could be the root cause of this issue?  I am not able to trace logs also.

 

Also, FYI, our FLINK cluster(4nodes) is setup on our Hadoop YARN cluster.

 

Thanks a lot.

 

 

Regards,

Raja.

[Raja.Aravapalli]

I have not passed visibly anything from my end, to application any path either HDFS or Local, that starts with “/tmp” ☹

 

 

Regards,

Raja.

 

From: miki haiat <[hidden email]>
Date: Tuesday, September 11, 2018 at 12:44 PM
To: Raja Aravapalli <[hidden email]>
Cc: user <[hidden email]>
Subject: [EXTERNAL] Re: Server side error: Cannot find required BLOB at /tmp/blobStore

 

/tmp/blobStore 

Is it the path  for checkpoints/savepoints  storage ?

 

On Tue, 11 Sep 2018, 10:01 Raja.Aravapalli, <[hidden email]> wrote:

Hi,

 

My Flink application which reads from Kafka and writes to HDFS is failing repeatedly with below exception:

 

Caused by: java.io.IOException: Server side error: Cannot find required BLOB at /tmp/blobStore-**********

 

Can someone please help me on, what could be the root cause of this issue?  I am not able to trace logs also.

 

Also, FYI, our FLINK cluster(4nodes) is setup on our Hadoop YARN cluster.

 

Thanks a lot.

 

 

Regards,

Raja.

[Lasse Nedergaard]

Hi.

From my presentation on Flink forward you can validate this

•We used EMR on Amazon’s Linux AMI

•We didn't change the default blob server location (/tmp)

•Default a cron job cleaning up in /tmp

•Solution change blob server location with blob.storage.directory

Den tir. 11. sep. 2018 kl. 09.14 skrev miki haiat <[hidden email]>:

/tmp/blobStore 

Is it the path  for checkpoints/savepoints  storage ?

On Tue, 11 Sep 2018, 10:01 Raja.Aravapalli, <[hidden email]> wrote:

Hi,

 

My Flink application which reads from Kafka and writes to HDFS is failing repeatedly with below exception:

 

Caused by: java.io.IOException: Server side error: Cannot find required BLOB at /tmp/blobStore-**********

 

Can someone please help me on, what could be the root cause of this issue?  I am not able to trace logs also.

 

Also, FYI, our FLINK cluster(4nodes) is setup on our Hadoop YARN cluster.

 

Thanks a lot.

 

 

Regards,

Raja.

[Raja.Aravapalli]

 

Thanks a lot Lasse.

 

 

So, the recommendation is to update the property “blob.storage.directory”, to some maintained path, other than “/tmp”, so the default cleanup will be eliminated ?

 

Also can you pls confirm if this is a hdfs path or a Local unix path ?

Please acknowledge if I have understood is correctly?

 

Thanks a lot again.

 

 

Regards,

Raja.

 

From: Lasse Nedergaard <[hidden email]>
Date: Tuesday, September 11, 2018 at 12:52 PM
To: miki haiat <[hidden email]>
Cc: Raja Aravapalli <[hidden email]>, user <[hidden email]>
Subject: [EXTERNAL] Re: Server side error: Cannot find required BLOB at /tmp/blobStore

 

Hi.

 

From my presentation on Flink forward you can validate this

•We used EMR on Amazon’s Linux AMI

•We didn't change the default blob server location (/tmp)

•Default a cron job cleaning up in /tmp

•Solution change blob server location with blob.storage.directory

 

Den tir. 11. sep. 2018 kl. 09.14 skrev miki haiat <[hidden email]>:

/tmp/blobStore 

Is it the path  for checkpoints/savepoints  storage ?

 

On Tue, 11 Sep 2018, 10:01 Raja.Aravapalli, <[hidden email]> wrote:

Hi,

 

My Flink application which reads from Kafka and writes to HDFS is failing repeatedly with below exception:

 

Caused by: java.io.IOException: Server side error: Cannot find required BLOB at /tmp/blobStore-**********

 

Can someone please help me on, what could be the root cause of this issue?  I am not able to trace logs also.

 

Also, FYI, our FLINK cluster(4nodes) is setup on our Hadoop YARN cluster.

 

Thanks a lot.

 

 

Regards,

Raja.