Flink Application JAR

[Halfon, Roey]

Hi All,
We have an application which contains few modules (one for the main application with the "driver" and few other for the operators implementation, configuration etc.).
At the moment we are building fat jar contains all the modules and all the 3^rd parties dependencies.

As we continue develop, the jar will grow and grow till it will be heavy.

Is there any recommendation for using our libraries? Maybe to use them as dependencies to our application? Does flink have any capability of dynamically update for 3^rd party jars?

 

How do you build your application?

Thanks,
Roey.  

[Timothy Victor]

I think any jars in $FLINK_HOME/lib gets picked up in the class path.

As for dynamic update, wouldn't you want to run it through your tests/CI before deploying?   

FWIW we also use a fat jar.   Its large but also since we just build a docker container anyway it doesn't really matter.

Tim

On Wed, Jul 3, 2019, 5:39 AM Halfon, Roey <[hidden email]> wrote:

Hi All,
We have an application which contains few modules (one for the main application with the "driver" and few other for the operators implementation, configuration etc.).
At the moment we are building fat jar contains all the modules and all the 3^rd parties dependencies.

As we continue develop, the jar will grow and grow till it will be heavy.

Is there any recommendation for using our libraries? Maybe to use them as dependencies to our application? Does flink have any capability of dynamically update for 3^rd party jars?

 

How do you build your application?

Thanks,
Roey.  

[Halfon, Roey]

Hi,
Thanks for answering,
Yes, Of course I would want to test it.
But after testing, I want to be able to upload only the changed/related jars and not all the others that weren't changed at all.. At the moment, for every code change in any part of the project, we need to deploy and test the whole project code. And we want to work in a more modular way.
As you said, I also think the size of the jar is less important than what I wrote above.

Thanks.

From: Timothy Victor <[hidden email]>
Sent: Wednesday, July 3, 2019 2:31 PM
To: Halfon, Roey <[hidden email]>
Cc: user <[hidden email]>; Cohen, Inbal <[hidden email]>; Dvir, Liran <[hidden email]>; Hasenfeld, Guy <[hidden email]>; Ochayon, Matan <[hidden email]>; Vidal, Gilad <[hidden email]>; Yehuda, Ben <[hidden email]>
Subject: Re: Flink Application JAR

 

I think any jars in $FLINK_HOME/lib gets picked up in the class path.

 

As for dynamic update, wouldn't you want to run it through your tests/CI before deploying?   

 

FWIW we also use a fat jar.   Its large but also since we just build a docker container anyway it doesn't really matter.

 

Tim

 

On Wed, Jul 3, 2019, 5:39 AM Halfon, Roey <[hidden email]> wrote:

Hi All,
We have an application which contains few modules (one for the main application with the "driver" and few other for the operators implementation, configuration etc.).
At the moment we are building fat jar contains all the modules and all the 3^rd parties dependencies.

As we continue develop, the jar will grow and grow till it will be heavy.

Is there any recommendation for using our libraries? Maybe to use them as dependencies to our application? Does flink have any capability of dynamically update for 3^rd party jars?

 

How do you build your application?

Thanks,
Roey.  

[Timothy Victor]

>> At the moment, for every code change in any part of the project, we need to deploy and test the whole project code

This is exactly what we do and want to do.   I rather have my build/test CI tell me if I broke something than keep a mental model of where a change may have impact.   It also makes our versioning and rollbacks easier since our whole codebase has a single version baked into to jar meta info.

The downside is a longer build cycle.  But there are some good build tools like Bazel to help reduce that time.  My 2 cents.

Tim

On Wed, Jul 3, 2019, 6:49 AM Halfon, Roey <[hidden email]> wrote:

Hi,
Thanks for answering,
Yes, Of course I would want to test it.
But after testing, I want to be able to upload only the changed/related jars and not all the others that weren't changed at all.. At the moment, for every code change in any part of the project, we need to deploy and test the whole project code. And we want to work in a more modular way.
As you said, I also think the size of the jar is less important than what I wrote above.

Thanks.

From: Timothy Victor <[hidden email]>
Sent: Wednesday, July 3, 2019 2:31 PM
To: Halfon, Roey <[hidden email]>
Cc: user <[hidden email]>; Cohen, Inbal <[hidden email]>; Dvir, Liran <[hidden email]>; Hasenfeld, Guy <[hidden email]>; Ochayon, Matan <[hidden email]>; Vidal, Gilad <[hidden email]>; Yehuda, Ben <[hidden email]>
Subject: Re: Flink Application JAR

 

I think any jars in $FLINK_HOME/lib gets picked up in the class path.

 

As for dynamic update, wouldn't you want to run it through your tests/CI before deploying?   

 

FWIW we also use a fat jar.   Its large but also since we just build a docker container anyway it doesn't really matter.

 

Tim

 

On Wed, Jul 3, 2019, 5:39 AM Halfon, Roey <[hidden email]> wrote:

Hi All,
We have an application which contains few modules (one for the main application with the "driver" and few other for the operators implementation, configuration etc.).
At the moment we are building fat jar contains all the modules and all the 3^rd parties dependencies.

As we continue develop, the jar will grow and grow till it will be heavy.

Is there any recommendation for using our libraries? Maybe to use them as dependencies to our application? Does flink have any capability of dynamically update for 3^rd party jars?

 

How do you build your application?

Thanks,
Roey.