Any change in behavior related to the "web.upload.dir" behavior between Flink 1.9 and 1.11

[Avijit Saha]

Hello,

Has there been any change in behavior related to the "web.upload.dir" behavior between Flink 1.9 and 1.11?

I have a failure case where when build an image using "flink:1.11.0-scala_2.12" in Dockerfile, the job manager job submissions fail with the following Exception but the same flow works fine (for the same underlying Code image) when using "flink:1.9.1-scala_2.12"..............

This is the Exception stack trace for 1.11 and not seen using 1.9:
------------------------------------------------------------------------------------------
Caused by: java.nio.file.FileAlreadyExistsException: /opt/flink/flink-web-upload
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88) ~[?:1.8.0_262]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_262]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_262]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_262]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_262]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_262]
        at java.nio.file.Files.createDirectories(Files.java:727) ~[?:1.8.0_262]
        at org.apache.flink.runtime.rest.RestServerEndpoint.checkAndCreateUploadDir(RestServerEndpoint.java:478) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.rest.RestServerEndpoint.createUploadDir(RestServerEndpoint.java:462) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.rest.RestServerEndpoint.<init>(RestServerEndpoint.java:114) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.webmonitor.WebMonitorEndpoint.<init>(WebMonitorEndpoint.java:200) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.<init>(DispatcherRestEndpoint.java:68) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.rest.SessionRestEndpointFactory.createRestEndpoint(SessionRestEndpointFactory.java:63) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:152) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:216) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        ... 2 more    

[Chesnay Schepler]

From what I can tell we have not changed anything.

Are you making any modifications to the image? This exception should only be thrown if there is already a file with the same path, and I don't think Flink would do that.

On 03/08/2020 21:43, Avijit Saha wrote:

Hello,

Has there been any change in behavior related to the "web.upload.dir" behavior between Flink 1.9 and 1.11?

I have a failure case where when build an image using "flink:1.11.0-scala_2.12" in Dockerfile, the job manager job submissions fail with the following Exception but the same flow works fine (for the same underlying Code image) when using "flink:1.9.1-scala_2.12"..............

This is the Exception stack trace for 1.11 and not seen using 1.9:
------------------------------------------------------------------------------------------
Caused by: java.nio.file.FileAlreadyExistsException: /opt/flink/flink-web-upload
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88) ~[?:1.8.0_262]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_262]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_262]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_262]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_262]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_262]
        at java.nio.file.Files.createDirectories(Files.java:727) ~[?:1.8.0_262]
        at org.apache.flink.runtime.rest.RestServerEndpoint.checkAndCreateUploadDir(RestServerEndpoint.java:478) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.rest.RestServerEndpoint.createUploadDir(RestServerEndpoint.java:462) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.rest.RestServerEndpoint.<init>(RestServerEndpoint.java:114) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.webmonitor.WebMonitorEndpoint.<init>(WebMonitorEndpoint.java:200) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.<init>(DispatcherRestEndpoint.java:68) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.rest.SessionRestEndpointFactory.createRestEndpoint(SessionRestEndpointFactory.java:63) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:152) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:216) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        ... 2 more    

[Avijit Saha]

Thanks!

It seems the problem went away when I started using 'ln -s $FLINK_HOME/usrlib $FLINK_HOME/flink-web-upload' in my Dockerfile!

On Mon, Aug 3, 2020 at 3:09 PM Chesnay Schepler <[hidden email]> wrote:

From what I can tell we have not changed anything.

Are you making any modifications to the image? This exception should only be thrown if there is already a file with the same path, and I don't think Flink would do that.

On 03/08/2020 21:43, Avijit Saha wrote:

Hello,

Has there been any change in behavior related to the "web.upload.dir" behavior between Flink 1.9 and 1.11?

I have a failure case where when build an image using "flink:1.11.0-scala_2.12" in Dockerfile, the job manager job submissions fail with the following Exception but the same flow works fine (for the same underlying Code image) when using "flink:1.9.1-scala_2.12"..............

This is the Exception stack trace for 1.11 and not seen using 1.9:
------------------------------------------------------------------------------------------
Caused by: java.nio.file.FileAlreadyExistsException: /opt/flink/flink-web-upload
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88) ~[?:1.8.0_262]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_262]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_262]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_262]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_262]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_262]
        at java.nio.file.Files.createDirectories(Files.java:727) ~[?:1.8.0_262]
        at org.apache.flink.runtime.rest.RestServerEndpoint.checkAndCreateUploadDir(RestServerEndpoint.java:478) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.rest.RestServerEndpoint.createUploadDir(RestServerEndpoint.java:462) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.rest.RestServerEndpoint.<init>(RestServerEndpoint.java:114) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.webmonitor.WebMonitorEndpoint.<init>(WebMonitorEndpoint.java:200) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.<init>(DispatcherRestEndpoint.java:68) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.rest.SessionRestEndpointFactory.createRestEndpoint(SessionRestEndpointFactory.java:63) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:152) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:216) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        ... 2 more