(DEPRECATED) Apache Flink User Mailing List archive.

Uploading job jar via web UI in flink HA mode

classic Classic list List threaded Threaded
♦
♦
 3 messages Options
sidhant gupta
Reply | Threaded
Open this post in threaded view
♦
♦
|

Uploading job jar via web UI in flink HA mode

sidhant gupta
Hi All,

I have 2 job managers in flink HA mode cluster setup. I have a load balancer forwarding request to both (leader and stand by) the job managers in default round-robin fashion. While uploading the job jar the Web UI is fluctuating between the leader and standby page. Its difficult to upload the job jar and also getting errors after submitting the job as below: 

Caused by: org.apache.flink.runtime.rest.handler.RestHandlerException: Jar file /tmp/flink-web-4be8d72f-6a7d-4ec9-b71e-516e58c629d5/flink-web-upload/ce291201-31e4-4e55-a68b-eae1a770518d_flinkfhirmapperjob-1.2.jar does not exist


I tried using web.upload.dir property to use a folder location in S3 but getting below error: 

2020-12-02 08:26:06,242 WARN  org.apache.flink.runtime.rest.FileUploadHandler              [] - File upload failed.

java.lang.IllegalArgumentException: UploadDirectory is not absolute.

    at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.runtime.rest.handler.FileUploads.<init>(FileUploads.java:58) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:159) [flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:68) [flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [flink-dist_2.11-1.11.2.jar:1.11.2]


Looks like web.upload.dir only supports local path.


Any suggestions on how to upload and submit job jar in flink HA cluster setup via web UI and also it from the CLI ?


Thanks and regards

Sidhant Gupta
Till Rohrmann
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Uploading job jar via web UI in flink HA mode

Till Rohrmann
Hi Sidhant,

Have you seen this discussion [1]? If you want to use S3, then you need to make sure that you start your Flink processes with the appropriate FileSystemProvider for S3 [2]. So the problem you are seeing is most likely caused by the JVM not knowing a S3 file system implementation.

Be aware that the filesystem used by the FileUploadHandler is java.nio.file.FileSystem and not Flink's org.apache.flink.core.fs.FileSystem for which we provide different FileSystem implementations.

My best recommendation would be to not use the web submission for Flink jobs when running Flink behind a load balancer. Instead use the CLI which will retrieve the leader address from ZooKeeper.


Cheers,
Till

On Wed, Dec 2, 2020 at 11:31 AM sidhant gupta <[hidden email]> wrote:
Hi All,

I have 2 job managers in flink HA mode cluster setup. I have a load balancer forwarding request to both (leader and stand by) the job managers in default round-robin fashion. While uploading the job jar the Web UI is fluctuating between the leader and standby page. Its difficult to upload the job jar and also getting errors after submitting the job as below: 

Caused by: org.apache.flink.runtime.rest.handler.RestHandlerException: Jar file /tmp/flink-web-4be8d72f-6a7d-4ec9-b71e-516e58c629d5/flink-web-upload/ce291201-31e4-4e55-a68b-eae1a770518d_flinkfhirmapperjob-1.2.jar does not exist


I tried using web.upload.dir property to use a folder location in S3 but getting below error: 

2020-12-02 08:26:06,242 WARN  org.apache.flink.runtime.rest.FileUploadHandler              [] - File upload failed.

java.lang.IllegalArgumentException: UploadDirectory is not absolute.

    at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.runtime.rest.handler.FileUploads.<init>(FileUploads.java:58) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:159) [flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:68) [flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [flink-dist_2.11-1.11.2.jar:1.11.2]


Looks like web.upload.dir only supports local path.


Any suggestions on how to upload and submit job jar in flink HA cluster setup via web UI and also it from the CLI ?


Thanks and regards

Sidhant Gupta
sidhant gupta
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Uploading job jar via web UI in flink HA mode

sidhant gupta
Hi Till,

Thanks for the clarification and suggestions

Regards
Sidhant Gupta

On Wed, Dec 2, 2020, 10:10 PM Till Rohrmann <[hidden email]> wrote:
Hi Sidhant,

Have you seen this discussion [1]? If you want to use S3, then you need to make sure that you start your Flink processes with the appropriate FileSystemProvider for S3 [2]. So the problem you are seeing is most likely caused by the JVM not knowing a S3 file system implementation.

Be aware that the filesystem used by the FileUploadHandler is java.nio.file.FileSystem and not Flink's org.apache.flink.core.fs.FileSystem for which we provide different FileSystem implementations.

My best recommendation would be to not use the web submission for Flink jobs when running Flink behind a load balancer. Instead use the CLI which will retrieve the leader address from ZooKeeper.


Cheers,
Till

On Wed, Dec 2, 2020 at 11:31 AM sidhant gupta <[hidden email]> wrote:
Hi All,

I have 2 job managers in flink HA mode cluster setup. I have a load balancer forwarding request to both (leader and stand by) the job managers in default round-robin fashion. While uploading the job jar the Web UI is fluctuating between the leader and standby page. Its difficult to upload the job jar and also getting errors after submitting the job as below: 

Caused by: org.apache.flink.runtime.rest.handler.RestHandlerException: Jar file /tmp/flink-web-4be8d72f-6a7d-4ec9-b71e-516e58c629d5/flink-web-upload/ce291201-31e4-4e55-a68b-eae1a770518d_flinkfhirmapperjob-1.2.jar does not exist


I tried using web.upload.dir property to use a folder location in S3 but getting below error: 

2020-12-02 08:26:06,242 WARN  org.apache.flink.runtime.rest.FileUploadHandler              [] - File upload failed.

java.lang.IllegalArgumentException: UploadDirectory is not absolute.

    at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.runtime.rest.handler.FileUploads.<init>(FileUploads.java:58) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:159) [flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:68) [flink-dist_2.11-1.11.2.jar:1.11.2]

    at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [flink-dist_2.11-1.11.2.jar:1.11.2]


Looks like web.upload.dir only supports local path.


Any suggestions on how to upload and submit job jar in flink HA cluster setup via web UI and also it from the CLI ?


Thanks and regards

Sidhant Gupta
Free forum by Nabble Edit this page