Flink SQL client always cost 2 minutes to submit job to a local cluster

[yinhua.dai]

I am using Flink 1.6.1, I tried to use flink sql client with some own jars
with --jar and --library.
It can work to execute sql query, however it always cause around 2 minutes
to submit the job the local cluster, but when I copy my jar to flink lib,
and remove --jar and --library parameter, it can submit the job immediately.

I debugged and found the 2 minutes is cost by the RestClusterClient to send
the request with the jar as payload to the flink cluster.

I don't know why it uses 2 minutes to upload the package? Is there a way to
work around it?
Thanks.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Hequn Cheng]

Hi, yinhua

Thanks for looking into the problem. I'm not familiar with the code of these part. As a workaround, you can put your jars into the flink lib folder or add your jars into the classpath. Hope this helps.

Best, Hequn

On Fri, Dec 28, 2018 at 11:52 AM yinhua.dai <[hidden email]> wrote:

I am using Flink 1.6.1, I tried to use flink sql client with some own jars
with --jar and --library.
It can work to execute sql query, however it always cause around 2 minutes
to submit the job the local cluster, but when I copy my jar to flink lib,
and remove --jar and --library parameter, it can submit the job immediately.

I debugged and found the 2 minutes is cost by the RestClusterClient to send
the request with the jar as payload to the flink cluster.

I don't know why it uses 2 minutes to upload the package? Is there a way to
work around it?
Thanks.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[yinhua.dai]

Hi Hequn,

Thanks.
I could do that for now, however I have to find another way because the jar
some times get update and the flink cluster will be remotely in future.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Fabian Hueske-2]

Hi,

You can try to build a JAR file with all runtime dependencies of Flink SQL (Calcite, Janino, +transitive dependencies), add it to the lib folder, and exclude the dependencies from the JAR file that is sent to the cluster when you submit a job.

It would also be good to figure out what takes so long.

Is it actually the submission of the jar file (in that case Hequn's and my suggestions would help) or the time to plan a query?

Best, Fabian

Am So., 30. Dez. 2018 um 22:06 Uhr schrieb yinhua.dai <[hidden email]>:

I have to do that for now, however I have to find another way because the jar
some times get update and the flink cluster will be remotely in future.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[yinhua.dai]

Hi Fabian,

It's the submission of the jar file cost too long time.
And yes Hequn and your suggestion is working, but just curious why a 100M
jar files causes so long time to submit, is it related with some upload
parameter settings of the web layer?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Hequn Cheng]

Hi yinhua,

Could you help to reproduce the problem? I can help to figure out the root cause. 

Best, Hequn

On Fri, Jan 4, 2019 at 11:37 AM yinhua.dai <[hidden email]> wrote:

Hi Fabian,

It's the submission of the jar file cost too long time.
And yes Hequn and your suggestion is working, but just curious why a 100M
jar files causes so long time to submit, is it related with some upload
parameter settings of the web layer?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/