Blobstore exceptions.

[Lasse Nedergaard]

Hi.

We sometimes see job fails with a blob store exception, like the one below. Anyone has an idea why we get them, and how to avoid them?.

In this case the job has run without any problems for a week and then we get the error. Only this job are affected right now all other running as expected and next time it can be one of the other jobs that get the exception. 

We running Flink 1.4.2, on AWS EMR cluster, but we have seen the same problems on 1.3.2 too.

Anyone 

java.io.IOException: Failed to fetch BLOB ff5d324719fb4caf3a0dba3fbcfa795e/p-812d84ea013302dbd24da1d32e732cc01582dabc-3198b6f63d293d2756f4cf5b8eebe7a2 from ip-10-1-1-192.eu-west-1.compute.internal/10.1.1.192:46781 and store it under /tmp/blobStore-3e90d7b0-2f40-4e28-b2b0-01d9ba96ac55/incoming/temp-00000173
	at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:191)
	at org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:177)
	at org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:205)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:119)
	at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:878)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:589)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: GET operation failed: Server side error: /tmp/blobStore-a83b8ca6-c01a-496a-8997-31687f37b95d/incoming/temp-00049050
	at org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:253)
	at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:166)
	... 6 more
Caused by: java.io.IOException: Server side error: /tmp/blobStore-a83b8ca6-c01a-496a-8997-31687f37b95d/incoming/temp-00049050
	at org.apache.flink.runtime.blob.BlobClient.receiveAndCheckGetResponse(BlobClient.java:306)
	at org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:247)
	... 7 more
Caused by: java.nio.file.NoSuchFileException: /tmp/blobStore-a83b8ca6-c01a-496a-8997-31687f37b95d/incoming/temp-00049050
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:452)
	at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:521)
	at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
	at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)

[Renjie Liu]

You need to check that whether you disk is full.

On Thu, Jun 14, 2018 at 2:15 PM Lasse Nedergaard <[hidden email]> wrote:

Hi.

We sometimes see job fails with a blob store exception, like the one below. Anyone has an idea why we get them, and how to avoid them?.

In this case the job has run without any problems for a week and then we get the error. Only this job are affected right now all other running as expected and next time it can be one of the other jobs that get the exception. 

We running Flink 1.4.2, on AWS EMR cluster, but we have seen the same problems on 1.3.2 too.

Anyone 

java.io.IOException: Failed to fetch BLOB ff5d324719fb4caf3a0dba3fbcfa795e/p-812d84ea013302dbd24da1d32e732cc01582dabc-3198b6f63d293d2756f4cf5b8eebe7a2 from ip-10-1-1-192.eu-west-1.compute.internal/10.1.1.192:46781 and store it under /tmp/blobStore-3e90d7b0-2f40-4e28-b2b0-01d9ba96ac55/incoming/temp-00000173
	at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:191)
	at org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:177)
	at org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:205)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:119)
	at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:878)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:589)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: GET operation failed: Server side error: /tmp/blobStore-a83b8ca6-c01a-496a-8997-31687f37b95d/incoming/temp-00049050
	at org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:253)
	at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:166)
	... 6 more
Caused by: java.io.IOException: Server side error: /tmp/blobStore-a83b8ca6-c01a-496a-8997-31687f37b95d/incoming/temp-00049050
	at org.apache.flink.runtime.blob.BlobClient.receiveAndCheckGetResponse(BlobClient.java:306)
	at org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:247)
	... 7 more
Caused by: java.nio.file.NoSuchFileException: /tmp/blobStore-a83b8ca6-c01a-496a-8997-31687f37b95d/incoming/temp-00049050
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:452)
	at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:521)
	at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
	at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)

--