User jar is not present in the flink job manager's class path

[yinhua.dai]

We have some customized log4j layout implementation so we need flink job
manager/task manager be able to load the logger implementation which is
packaged in the uber jar.

However, we noticed that in flink 1.3, the user jar is put at the beginning
of job manager, when we do the same again in flink 1.5, the user jar is not
there any more.
Is this expected?

I saw this is the document:
*When submitting a Flink job/application directly to YARN (via bin/flink run
-m yarn-cluster ...), dedicated TaskManagers and JobManagers are started for
that job. Those JVMs have both Flink framework classes and user code classes
in the Java classpath. That means that there is no dynamic classloading
involved in that case.*

And we are using flink on yarn with per-job mode.
So confused by what we experiencing for now.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Timo Walther]

Hi,

did you try to change the classloading strategy? Maybe this problem
could be fixed by configuring the ClassLoader resolution order [1].

Regards,
Timo

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/debugging_classloading.html

Am 11.10.18 um 10:49 schrieb yinhua.dai:

> We have some customized log4j layout implementation so we need flink job
> manager/task manager be able to load the logger implementation which is
> packaged in the uber jar.
>
> However, we noticed that in flink 1.3, the user jar is put at the beginning
> of job manager, when we do the same again in flink 1.5, the user jar is not
> there any more.
> Is this expected?
>
> I saw this is the document:
> *When submitting a Flink job/application directly to YARN (via bin/flink run
> -m yarn-cluster ...), dedicated TaskManagers and JobManagers are started for
> that job. Those JVMs have both Flink framework classes and user code classes
> in the Java classpath. That means that there is no dynamic classloading
> involved in that case.*
>
> And we are using flink on yarn with per-job mode.
> So confused by what we experiencing for now.
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[yinhua.dai]

Hi Timo,

I didn't tried to configure the classloader order, according to the
document, it should only be needed for yarn-session mode, right?
I assume "child-first" should be better for general use case for flink job, right?

I can see the ship files(-yt /path/dir/) is present in job manager's class
path, so maybe I should put my uber jar in the -yt path so that it will be
shipped and add to class path in flink 1.5?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Timo Walther]

Yes, you are right. I was not aware that the resolution order depends on
the cluster deployment. I will loop in Gary (in CC) that might know
about such a YARN setup.

Regards,
Timo

Am 11.10.18 um 15:47 schrieb yinhua.dai:

> Hi Timo,
>
> I didn't tried to configure the classloader order, according to the
> document, it should only be needed for yarn-session mode, right?
>
> I can see the ship files(-yt /path/dir/) is present in job manager's class
> path, so maybe I should put my uber jar in the -yt path so that it will be
> shipped and add to class path in flink 1.5?
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[yinhua.dai]

Meanwhile, I can see below code in flink 1.5

public static final ConfigOption<String> CLASSPATH_INCLUDE_USER_JAR =
                key("yarn.per-job-cluster.include-user-jar")
                        .defaultValue("ORDER")
                        .withDescription("Defines whether user-jars are included in the system
class path for per-job-clusters as" +
                                " well as their positioning in the path. They can be positioned at the
beginning (\"FIRST\"), at the" +
                                " end (\"LAST\"), or be positioned based on their name (\"ORDER\").
Setting this parameter to" +
                                " \"DISABLED\" causes the jar to be included in the user class path
instead.");

Does this mean the user jar should always be included in class path?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Gary Yao-2]

Hi,

Could it be that you are submitting the job in attached mode, i.e., without -d
parameter? In the "job cluster attached mode", we actually start a Flink
session cluster (and stop it again from the CLI) [1]. Therefore, in attached
mode, the config option "yarn.per-job-cluster.include-user-jar" is effectively
ignored. If you submit with -d, a "true job cluster" is started, and the user
jar should be added to the system classpath. Alternatively, if the detached
mode is not an option for you, you could add a jar with your custom logger
implementation to the flink /lib directory.

If the behavior in Flink 1.3 is indeed different, then I would consider this a
regression. Can you open a jira issue for that?

Best,
Gary

[1] https://github.com/apache/flink/blob/d13015e23c805e1aaefdc7d8037f2fa87ea74830/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java#L261

On Thu, Oct 11, 2018 at 3:54 PM Timo Walther <[hidden email]> wrote:

Yes, you are right. I was not aware that the resolution order depends on
the cluster deployment. I will loop in Gary (in CC) that might know
about such a YARN setup.

Regards,
Timo

Am 11.10.18 um 15:47 schrieb yinhua.dai:
> Hi Timo,
>
> I didn't tried to configure the classloader order, according to the
> document, it should only be needed for yarn-session mode, right?
>
> I can see the ship files(-yt /path/dir/) is present in job manager's class
> path, so maybe I should put my uber jar in the -yt path so that it will be
> shipped and add to class path in flink 1.5?
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[yinhua.dai]

Hi Gary,

Yes you are right, we are using the attach mode.
I will try to put my jar to flink/lib to get around with the issue.
Thanks.

https://issues.apache.org/jira/browse/FLINK-10535 is raised for the discrepancy for flink 1.3 and 1.5, thanks a lot.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/