Running Apache Flink on the GraalVM as a Native Image

Whats up guys,

 

I'm trying to run an Apache Flink Application with the GraalVM Native Image but I get the following error: (check attached file)

 

I suppose this happens, because Flink uses a lot of low-level-code and is highly optimized.

 

When I googled the combination of GraalVM Native Image and Apache Flink I get no results.

 

Did anyone ever succeeded in making it work and how?

 

Best regards,

 

Ivo



[Stephen Connolly]

On Thu 25 Jun 2020 at 12:48, [hidden email] <[hidden email]> wrote:

Whats up guys,

 

I'm trying to run an Apache Flink Application with the GraalVM Native Image but I get the following error: (check attached file)

 

I suppose this happens, because Flink uses a lot of low-level-code and is highly optimized.

Actually I suspect the reason is that Flink uses dynamic classloading.

GraalVM requires all the code available in order to produce a native image.

You’d need to pre-bind the topology you want Flink to run into the native image.

More fun, you’ll actually need two images, one for the job manager and one for the task manager.

And you’ll need to convince GraalVM that the entry-point is your topology needs reflection support enabled... plus whatever other classes use reflection in Flink.

Sounds rather complex to me. If native images are what is important to you, there seemed to be a strong contender in the Rust language community, didn’t provide as strong management as Flink, and you’d probably have more work managing things like checkpointing, but if native code is important that’s where I’d be looking. Sadly I cannot remember the name and my google-foo is weak tonight

 

When I googled the combination of GraalVM Native Image and Apache Flink I get no results.

 

Did anyone ever succeeded in making it work and how?

 

Best regards,

 

Ivo



--

[Stephen Connolly]

On Sun 28 Jun 2020 at 01:34, Stephen Connolly <[hidden email]> wrote:

On Thu 25 Jun 2020 at 12:48, [hidden email] <[hidden email]> wrote:

Whats up guys,

 

I'm trying to run an Apache Flink Application with the GraalVM Native Image but I get the following error: (check attached file)

 

I suppose this happens, because Flink uses a lot of low-level-code and is highly optimized.

Actually I suspect the reason is that Flink uses dynamic classloading.

GraalVM requires all the code available in order to produce a native image.

You’d need to pre-bind the topology you want Flink to run into the native image.

More fun, you’ll actually need two images, one for the job manager and one for the task manager.

And you’ll need to convince GraalVM that the entry-point is your topology needs reflection support enabled... plus whatever other classes use reflection in Flink.

Sounds rather complex to me. If native images are what is important to you, there seemed to be a strong contender in the Rust language community, didn’t provide as strong management as Flink, and you’d probably have more work managing things like checkpointing, but if native code is important that’s where I’d be looking. Sadly I cannot remember the name and my google-foo is weak tonight

I think it might have been 

https://github.com/grippy/tempest but that looks less actively developed than the one I thought I saw...

I’d also check out frameworks for Go even if I dislike Go... if you want native code it’s either Rust or Go in my book

 

When I googled the combination of GraalVM Native Image and Apache Flink I get no results.

 

Did anyone ever succeeded in making it work and how?

 

Best regards,

 

Ivo



--

--

[Roman Grebennikov]

Hi,

the error it the original message means that inside a Kryo serializer, there is something that uses sun.misc.Unsafe to compute field offset in raw memory to access class fields. The problem is that memory layout in JVM and in NI is different and most probably will result in a segfault if you leave it as is.

If you really want to go further, there is a @RecomputeFieldValue(kind = Kind.FieldOffset) annotation in the graalvm to update the offset for NI.

But I guess the overall perspective of running Flink as a NI is not that good for the following reasons:

* GraalVM NI requires that the whole application is completely static and there is no dynamic classloading happening in runtime. NI still can handle some sort of dynamic classloading, but only if you define all possible combinations of loaded classes on compile time. Flink itself uses dynamic classloading when you submit a job there as a fat jar file. So you will need to statically compile a bundle of your app fat jar AND taskmanager code at once into a single binary.

* Flink also relies quite a lot on dynamic reflection in the serialization code, so you also have to build static reflection configuration at compile-time (by probably running your job+taskmanager with tracing agent on JVM)

* As GraalVM NI has no way of doing JIT compilation (as everything must be compiled statically), you should expect a lower overall job performance.

GraalVM and Flink can be quite a nice code gymnastics exercise, but what is your final business goal?

Roman Grebennikov | [hidden email]

On Thu, Jun 25, 2020, at 11:48, [hidden email] wrote:

Whats up guys,

 

I'm trying to run an Apache Flink Application with the GraalVM Native Image but I get the following error: (check attached file)

 

I suppose this happens, because Flink uses a lot of low-level-code and is highly optimized.

 

When I googled the combination of GraalVM Native Image and Apache Flink I get no results.

 

Did anyone ever succeeded in making it work and how?

 

Best regards,

 

Ivo



Attachments:

• Flink GraalVM Native Image Error
  

[Arvid Heise-3]

Using Flink in a NI does not make much sense. However, it's definitely worthwhile to enable Flink applications to use Flink as a library and bundle that in a NI.

Btw reflection is not the root problem. You can use reflection just fine in a NI. The issue with NI is (next to Unsafe) that NI has a close-world assumption. That means we would only run Flink without any UDFs (in the sense of real user code, not UDFs provided by Flink). However, in theory, it should be possible to compile Flink against a specific application and build a NI when everything that is needed for a particular Flink application is known to the Graal NI compiler. In practise, Flink would need that compiler to figure out, which classes/methods are actually needed and which not, since some Flink magic would break the close world assumption.

Finally, Table API with code generation will probably not work with Graal for the time being at all. Not sure if there is any effort to support that as well on Graal side.

On Tue, Jun 30, 2020 at 9:51 AM Roman Grebennikov <[hidden email]> wrote:

Hi,

the error it the original message means that inside a Kryo serializer, there is something that uses sun.misc.Unsafe to compute field offset in raw memory to access class fields. The problem is that memory layout in JVM and in NI is different and most probably will result in a segfault if you leave it as is.

If you really want to go further, there is a @RecomputeFieldValue(kind = Kind.FieldOffset) annotation in the graalvm to update the offset for NI.

But I guess the overall perspective of running Flink as a NI is not that good for the following reasons:

* GraalVM NI requires that the whole application is completely static and there is no dynamic classloading happening in runtime. NI still can handle some sort of dynamic classloading, but only if you define all possible combinations of loaded classes on compile time. Flink itself uses dynamic classloading when you submit a job there as a fat jar file. So you will need to statically compile a bundle of your app fat jar AND taskmanager code at once into a single binary.

* Flink also relies quite a lot on dynamic reflection in the serialization code, so you also have to build static reflection configuration at compile-time (by probably running your job+taskmanager with tracing agent on JVM)

* As GraalVM NI has no way of doing JIT compilation (as everything must be compiled statically), you should expect a lower overall job performance.

GraalVM and Flink can be quite a nice code gymnastics exercise, but what is your final business goal?

Roman Grebennikov | [hidden email]

On Thu, Jun 25, 2020, at 11:48, [hidden email] wrote:

Whats up guys,

 

I'm trying to run an Apache Flink Application with the GraalVM Native Image but I get the following error: (check attached file)

 

I suppose this happens, because Flink uses a lot of low-level-code and is highly optimized.

 

When I googled the combination of GraalVM Native Image and Apache Flink I get no results.

 

Did anyone ever succeeded in making it work and how?

 

Best regards,

 

Ivo



Attachments:

• Flink GraalVM Native Image Error
  

--