Flink and Akka version incompatibility ..

[Debasish Ghosh]

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

[Haibo Sun]

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

[Debasish Ghosh]

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

[tison]

I can see that we relocate akka's netty, akka uncommon math but also

be curious why Flink doesn't shaded all of akka dependencies...

Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

[Debasish Ghosh]

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:

I can see that we relocate akka's netty, akka uncommon math but also

be curious why Flink doesn't shaded all of akka dependencies...

Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

--

[Haibo Sun]

Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.

I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,

Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:

I can see that we relocate akka's netty, akka uncommon math but also

be curious why Flink doesn't shaded all of akka dependencies...

Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

--

[Jeff Zhang]

I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.

Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道：

Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.

I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,

Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:

I can see that we relocate akka's netty, akka uncommon math but also

be curious why Flink doesn't shaded all of akka dependencies...

Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

--

--

[Debasish Ghosh]

For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:

I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.

Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道：

Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.

I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,

Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:

I can see that we relocate akka's netty, akka uncommon math but also

be curious why Flink doesn't shaded all of akka dependencies...

Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

--

--

--

[Haibo Sun]

Hi  Debasish Ghosh,

I agree that Flink should shade its Akka. 

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,

Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <[hidden email]> wrote:

For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:

I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.

Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道：

Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.

I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,

Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:

I can see that we relocate akka's netty, akka uncommon math but also

be curious why Flink doesn't shaded all of akka dependencies...

Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

--

--

--

[Debasish Ghosh]

Hi Haibo - Thanks for the clarification ..

regards.

On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,

I agree that Flink should shade its Akka. 

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,

Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <[hidden email]> wrote:

For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:

I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.

Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道：

Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.

I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,

Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:

I can see that we relocate akka's netty, akka uncommon math but also

be curious why Flink doesn't shaded all of akka dependencies...

Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

--

--

--

--

[Debasish Ghosh]

Also wanted to check if anyone has ventured into this exercise of shading Akka in Flink .. 

Is this something that qualifies as one of the roadmap items in Flink ?

regards.

On Wed, Jul 24, 2019 at 3:44 PM Debasish Ghosh <[hidden email]> wrote:

Hi Haibo - Thanks for the clarification ..

regards.

On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,

I agree that Flink should shade its Akka. 

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,

Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <[hidden email]> wrote:

For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:

I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.

Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道：

Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.

I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,

Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:

I can see that we relocate akka's netty, akka uncommon math but also

be curious why Flink doesn't shaded all of akka dependencies...

Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

--

--

--

--

--

[Haibo Sun]

The following JIRA is about the problem you encounter. I think you should be very interested in its comments.There does seem to be a problem with shading Akka, and Flink is considering isolating the classloader that contain Akka and Scala to allow the applications and Flink to use different Akka versions.

https://issues.apache.org/jira/browse/FLINK-10903

Best,

Haibo

At 2019-07-25 00:07:27, "Debasish Ghosh" <[hidden email]> wrote:

Also wanted to check if anyone has ventured into this exercise of shading Akka in Flink .. 

Is this something that qualifies as one of the roadmap items in Flink ?

regards.

On Wed, Jul 24, 2019 at 3:44 PM Debasish Ghosh <[hidden email]> wrote:

Hi Haibo - Thanks for the clarification ..

regards.

On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,

I agree that Flink should shade its Akka. 

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,

Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <[hidden email]> wrote:

For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:

I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.

Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道：

Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.

I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,

Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:

I can see that we relocate akka's netty, akka uncommon math but also

be curious why Flink doesn't shaded all of akka dependencies...

Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

--

--

--

--

--

[Debasish Ghosh]

Thanks for the pointer .. yes, we need this ..

On Thu, Jul 25, 2019 at 9:08 AM Haibo Sun <[hidden email]> wrote:

The following JIRA is about the problem you encounter. I think you should be very interested in its comments.There does seem to be a problem with shading Akka, and Flink is considering isolating the classloader that contain Akka and Scala to allow the applications and Flink to use different Akka versions.

https://issues.apache.org/jira/browse/FLINK-10903

Best,

Haibo

At 2019-07-25 00:07:27, "Debasish Ghosh" <[hidden email]> wrote:

Also wanted to check if anyone has ventured into this exercise of shading Akka in Flink .. 

Is this something that qualifies as one of the roadmap items in Flink ?

regards.

On Wed, Jul 24, 2019 at 3:44 PM Debasish Ghosh <[hidden email]> wrote:

Hi Haibo - Thanks for the clarification ..

regards.

On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,

I agree that Flink should shade its Akka. 

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,

Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <[hidden email]> wrote:

For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:

I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.

Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道：

Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.

I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,

Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:

The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:

I can see that we relocate akka's netty, akka uncommon math but also

be curious why Flink doesn't shaded all of akka dependencies...

Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道：

Hello Haibo -

Yes, my application depends on Akka 2.5. 

Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:

Hi  Debasish Ghosh,
 

Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 

If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,

Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:

Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

regards.

--

--

--

--

--

--

--

--