JM upload files to blob server is slow

[forideal]

Hello my friend：

    My line of code runs very slowly. What are the possibilities?

 code：

 CompletableFuture<JobGraph> jarUploadFuture = jobGraphFuture.thenCombine(blobServerPortFuture, (jobGraph, blobServerPort) -> {
  final InetSocketAddress address = new InetSocketAddress(gateway.getHostname(), blobServerPort);
   try {
      ClientUtils.extractAndUploadJobGraphFiles(jobGraph, () -> new BlobClient(address, configuration));
   } catch (FlinkException e) {
      throw new CompletionException(e);
   }
   return jobGraph;
});

code link：https://github.com/apache/flink/blob/release-1.10.0/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/JarRunHandler.java#L93

Through jstack, we can also find that the thread Flink-DispatcherRestEndpoint-thread-1 has been running for 5 minutes.

"Flink-DispatcherRestEndpoint-thread-1" #82 daemon prio=5 os_prio=0 tid=0x00007f590c03c800 nid=0x179 runnable [0x00007f5a34165000]

   java.lang.Thread.State: RUNNABLE

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)

at java.net.SocketOutputStream.write(SocketOutputStream.java:155)

at org.apache.flink.runtime.blob.BlobOutputStream.write(BlobOutputStream.java:88)

at org.apache.flink.util.IOUtils.copyBytes(IOUtils.java:65)

at org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:368)

at org.apache.flink.runtime.blob.BlobClient.uploadFile(BlobClient.java:428)

at org.apache.flink.runtime.client.ClientUtils.uploadUserJars(ClientUtils.java:102)

at org.apache.flink.runtime.client.ClientUtils.uploadAndSetUserJars(ClientUtils.java:95)

at org.apache.flink.runtime.client.ClientUtils.uploadJobGraphFiles(ClientUtils.java:77)

at org.apache.flink.runtime.client.ClientUtils.extractAndUploadJobGraphFiles(ClientUtils.java:57)

at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(RedJarRunHandler.java:92)

at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler$$Lambda$770/941040494.apply(Unknown Source)

at java.util.concurrent.CompletableFuture.biApply(CompletableFuture.java:1105)

at java.util.concurrent.CompletableFuture$BiApply.tryFire(CompletableFuture.java:1070)

at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Thank you very much for your reply, forideal

 

[Arvid Heise-3]

A jar upload shouldn't take minutes. There are two possibilities that likely co-occured:

- your jar is much bigger than needed. Did you make sure that you don't put Flink into the fatjar? That's counterproductive on many levels. Please check the jar size.

- your connection to the JM is bad. Where is your Flink cluster running? If connection is superslow, it's often better to build the jar on the respective server and submit on command line or through REST.

On Wed, Nov 4, 2020 at 7:15 AM forideal <[hidden email]> wrote:

Hello my friend：

    My line of code runs very slowly. What are the possibilities?

 code：

 CompletableFuture<JobGraph> jarUploadFuture = jobGraphFuture.thenCombine(blobServerPortFuture, (jobGraph, blobServerPort) -> {
  final InetSocketAddress address = new InetSocketAddress(gateway.getHostname(), blobServerPort);
   try {
      ClientUtils.extractAndUploadJobGraphFiles(jobGraph, () -> new BlobClient(address, configuration));
   } catch (FlinkException e) {
      throw new CompletionException(e);
   }
   return jobGraph;
});

code link：https://github.com/apache/flink/blob/release-1.10.0/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/JarRunHandler.java#L93

Through jstack, we can also find that the thread Flink-DispatcherRestEndpoint-thread-1 has been running for 5 minutes.

"Flink-DispatcherRestEndpoint-thread-1" #82 daemon prio=5 os_prio=0 tid=0x00007f590c03c800 nid=0x179 runnable [0x00007f5a34165000]

   java.lang.Thread.State: RUNNABLE

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)

at java.net.SocketOutputStream.write(SocketOutputStream.java:155)

at org.apache.flink.runtime.blob.BlobOutputStream.write(BlobOutputStream.java:88)

at org.apache.flink.util.IOUtils.copyBytes(IOUtils.java:65)

at org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:368)

at org.apache.flink.runtime.blob.BlobClient.uploadFile(BlobClient.java:428)

at org.apache.flink.runtime.client.ClientUtils.uploadUserJars(ClientUtils.java:102)

at org.apache.flink.runtime.client.ClientUtils.uploadAndSetUserJars(ClientUtils.java:95)

at org.apache.flink.runtime.client.ClientUtils.uploadJobGraphFiles(ClientUtils.java:77)

at org.apache.flink.runtime.client.ClientUtils.extractAndUploadJobGraphFiles(ClientUtils.java:57)

at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(RedJarRunHandler.java:92)

at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler$$Lambda$770/941040494.apply(Unknown Source)

at java.util.concurrent.CompletableFuture.biApply(CompletableFuture.java:1105)

at java.util.concurrent.CompletableFuture$BiApply.tryFire(CompletableFuture.java:1070)

at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Thank you very much for your reply, forideal

 

--

[forideal]

Hi Arvid Heise,

         Thank you for your reply.

         Yes,my connection to the JM is bad !!!

Best wishes,forideal

At 2020-11-04 15:32:38, "Arvid Heise" <[hidden email]> wrote:

A jar upload shouldn't take minutes. There are two possibilities that likely co-occured:

- your jar is much bigger than needed. Did you make sure that you don't put Flink into the fatjar? That's counterproductive on many levels. Please check the jar size.

- your connection to the JM is bad. Where is your Flink cluster running? If connection is superslow, it's often better to build the jar on the respective server and submit on command line or through REST.

On Wed, Nov 4, 2020 at 7:15 AM forideal <[hidden email]> wrote:

Hello my friend：

    My line of code runs very slowly. What are the possibilities?

 code：

 CompletableFuture<JobGraph> jarUploadFuture = jobGraphFuture.thenCombine(blobServerPortFuture, (jobGraph, blobServerPort) -> {
  final InetSocketAddress address = new InetSocketAddress(gateway.getHostname(), blobServerPort);
   try {
      ClientUtils.extractAndUploadJobGraphFiles(jobGraph, () -> new BlobClient(address, configuration));
   } catch (FlinkException e) {
      throw new CompletionException(e);
   }
   return jobGraph;
});

code link：https://github.com/apache/flink/blob/release-1.10.0/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/JarRunHandler.java#L93

Through jstack, we can also find that the thread Flink-DispatcherRestEndpoint-thread-1 has been running for 5 minutes.

"Flink-DispatcherRestEndpoint-thread-1" #82 daemon prio=5 os_prio=0 tid=0x00007f590c03c800 nid=0x179 runnable [0x00007f5a34165000]

   java.lang.Thread.State: RUNNABLE

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)

at java.net.SocketOutputStream.write(SocketOutputStream.java:155)

at org.apache.flink.runtime.blob.BlobOutputStream.write(BlobOutputStream.java:88)

at org.apache.flink.util.IOUtils.copyBytes(IOUtils.java:65)

at org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:368)

at org.apache.flink.runtime.blob.BlobClient.uploadFile(BlobClient.java:428)

at org.apache.flink.runtime.client.ClientUtils.uploadUserJars(ClientUtils.java:102)

at org.apache.flink.runtime.client.ClientUtils.uploadAndSetUserJars(ClientUtils.java:95)

at org.apache.flink.runtime.client.ClientUtils.uploadJobGraphFiles(ClientUtils.java:77)

at org.apache.flink.runtime.client.ClientUtils.extractAndUploadJobGraphFiles(ClientUtils.java:57)

at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(RedJarRunHandler.java:92)

at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler$$Lambda$770/941040494.apply(Unknown Source)

at java.util.concurrent.CompletableFuture.biApply(CompletableFuture.java:1105)

at java.util.concurrent.CompletableFuture$BiApply.tryFire(CompletableFuture.java:1070)

at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Thank you very much for your reply, forideal

 

--