How to use thin JAR instead of fat JAR when submitting Flink job?

[Qi Kang]

Hi,

According to the documentation of Flink, it seems that fat JAR is recommended when submitting a Flink job. However, the Flink dependencies (as well as other dependencies like Hadoop) are too big in size, thus producing a fat JAR which exceeds 100MB. Is there some way to separate the 'common' dependencies and app-specific dependencies and submit a thinner JAR? Many thanks.

 

[Dian Fu]

Hi Qi Kang,

You don't need and also should not package the dependencies of Flink to the job jar. Only application specific dependencies are enough.

Regards,
Dian

> 在 2019年9月23日，下午5:17，Qi Kang <[hidden email]> 写道：
>
> Hi,
>
> According to the documentation of Flink, it seems that fat JAR is recommended when submitting a Flink job. However, the Flink dependencies (as well as other dependencies like Hadoop) are too big in size, thus producing a fat JAR which exceeds 100MB. Is there some way to separate the 'common' dependencies and app-specific dependencies and submit a thinner JAR? Many thanks.
>
>
>

[Fabian Hueske-2]

Hi,

To expand on Dian's answer.

You should not add Flink's core libraries (APIs, core, runtime, etc.) to your fat JAR. However, connector dependencies (like Kafka, Cassandra, etc.) should be added.

If all your jobs require the same dependencies, you can also add JAR files to the ./lib folder of your job and taskmanager.

All JARs in the ./lib folder will be loaded into the system class loader. Note, that this means that they are shared across all jobs running on a taskmanager which reduces the isolation of jobs.

Best, Fabian

Am Mo., 23. Sept. 2019 um 11:37 Uhr schrieb Dian Fu <[hidden email]>:

Hi Qi Kang,

You don't need and also should not package the dependencies of Flink to the job jar. Only application specific dependencies are enough.

Regards,
Dian

> 在 2019年9月23日，下午5:17，Qi Kang <[hidden email]> 写道：
>
> Hi,
>
> According to the documentation of Flink, it seems that fat JAR is recommended when submitting a Flink job. However, the Flink dependencies (as well as other dependencies like Hadoop) are too big in size, thus producing a fat JAR which exceeds 100MB. Is there some way to separate the 'common' dependencies and app-specific dependencies and submit a thinner JAR? Many thanks.
>
>
>