Re: Objects deserialization on Jobmanager

Posted by Stephan Ewen on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Objects-deserialization-on-Jobmanager-tp1104p1107.html

Hi Ventura!

Flink loads on startup only the Flink classes and all user code classes are loaded dynamically. Each point when a user class is used, or a user class object is deserialized, Flink uses the user code classloader of that specific job.

For you as the user, this has the implication that if try to load a class via "Class.forName", you need to supply the user code classloader as well.

You can grab the usercode class loader in various ways:

1) Inside a UDF, you can grab it by making this a RichFunction and using "getRuntimeContext().getUsercodeClassloader()"

2) At almost any other play, you can use "Thread.currentThread().getContextClassloader()"

Hope that helps.

Greetings,

Stephan

On Wed, Apr 22, 2015 at 4:48 PM, Till Rohrmann <[hidden email]> wrote:

The corresponding code snippet could also help.

Cheers,

Till

On Wed, Apr 22, 2015 at 4:45 PM, Robert Metzger <[hidden email]> wrote:

Hi,

which version of Flink are you using?

Can you send us the complete stack trace of the error to help us understand the exact location where the issue occurs?

On Wed, Apr 22, 2015 at 4:33 PM, Ventura Del Monte <[hidden email]> wrote:

Hello, I am working on a flink-based deep learning library for my master's thesis. I am experiencing this issue at the moment: I have a java class with a transient field, so I had to write both a kryo custom serializer and a java one. The (de)serialization needs to access another object of my system, so if I run my software locally it works fine because the needed object is instantiated meanwhile it crashes when I run it in a remote environment because when the jobmanager receives the data, the object needed for the deserialization is not present in the system. Thus, my question is whether it is possible to let the jobmanager execute some user code or would it be better to edit the architecture of my system in order to avoid this kind of problem?

Regards,

Ventura