Customizing the classpath for TM and JM

[Alexander Alexandrov]

Hi there,

is there a canonical / suggested way to customize the classpath of the Flink JM and TM processes?

At the moment I hardcoded my way around this by manually changing the following lines like that:

# in taskmanager.sh

$JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"

# in jobmanagre.sh

$JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"

Perhaps there is a variable that I can set in conf/flink.yaml that will do the same trick.

[Ufuk Celebi]

Hey Alexander,

there is only the option "env.java.opts", which can be set in the config. I think you cannot use this to have multiple "-classpath" options when starting Java?

It makes sense to have the option to add something to the classpath. I will check this and fix it. :-) What about "taskmanager.env.java.classpath" and "jobmanager.env.java.classpath" as a configuration key? And maybe an additional "env.java.classpath"?

PS: sorry for the incomplete mail before :D

On 15 Jan 2015, at 01:00, Alexander Alexandrov <[hidden email]> wrote:

> Hi there,
>
> is there a canonical / suggested way to customize the classpath of the Flink JM and TM processes?
>
> At the moment I hardcoded my way around this by manually changing the following lines like that:
>
> # in taskmanager.sh
> $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
>
> # in jobmanagre.sh
> $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
>
> Perhaps there is a variable that I can set in conf/flink.yaml that will do the same trick.

[Alexander Alexandrov]

No worries, this is the first reply that I get in this thread.

What would be the difference between the two? I can make a PR if we agree on that.

2015-01-15 11:28 GMT+01:00 Ufuk Celebi <[hidden email]>:

Hey Alexander,

there is only the option "env.java.opts", which can be set in the config. I think you cannot use this to have multiple "-classpath" options when starting Java?

It makes sense to have the option to add something to the classpath. I will check this and fix it. :-) What about "taskmanager.env.java.classpath" and "jobmanager.env.java.classpath" as a configuration key? And maybe an additional "env.java.classpath"?

PS: sorry for the incomplete mail before :D

On 15 Jan 2015, at 01:00, Alexander Alexandrov <[hidden email]> wrote:

> Hi there,
>
> is there a canonical / suggested way to customize the classpath of the Flink JM and TM processes?
>
> At the moment I hardcoded my way around this by manually changing the following lines like that:
>
> # in taskmanager.sh
> $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
>
> # in jobmanagre.sh
> $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
>
> Perhaps there is a variable that I can set in conf/flink.yaml that will do the same trick.

[Alexander Alexandrov]

Sorry, I meant between the three. In particular, where will env.java.classpath propagate?

2015-01-15 14:24 GMT+01:00 Alexander Alexandrov <[hidden email]>:

No worries, this is the first reply that I get in this thread.

What would be the difference between the two? I can make a PR if we agree on that.

2015-01-15 11:28 GMT+01:00 Ufuk Celebi <[hidden email]>:

Hey Alexander,

there is only the option "env.java.opts", which can be set in the config. I think you cannot use this to have multiple "-classpath" options when starting Java?

It makes sense to have the option to add something to the classpath. I will check this and fix it. :-) What about "taskmanager.env.java.classpath" and "jobmanager.env.java.classpath" as a configuration key? And maybe an additional "env.java.classpath"?

PS: sorry for the incomplete mail before :D

On 15 Jan 2015, at 01:00, Alexander Alexandrov <[hidden email]> wrote:

> Hi there,
>
> is there a canonical / suggested way to customize the classpath of the Flink JM and TM processes?
>
> At the moment I hardcoded my way around this by manually changing the following lines like that:
>
> # in taskmanager.sh
> $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
>
> # in jobmanagre.sh
> $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
>
> Perhaps there is a variable that I can set in conf/flink.yaml that will do the same trick.

[Ufuk Celebi]

"env.java.classpath" could be used for global class path settings (both JM and TM) and the other ones for JM and TM respectively. I don't know if this is really necessary though. Because no one else asked for it, I think it's fine to go with what suits your use case.

– Ufuk

On 15 Jan 2015, at 14:25, Alexander Alexandrov <[hidden email]> wrote:

> Sorry, I meant between the three. In particular, where will env.java.classpath propagate?
>
> 2015-01-15 14:24 GMT+01:00 Alexander Alexandrov <[hidden email]>:
> No worries, this is the first reply that I get in this thread.
>
> What would be the difference between the two? I can make a PR if we agree on that.
>
> 2015-01-15 11:28 GMT+01:00 Ufuk Celebi <[hidden email]>:
> Hey Alexander,
>
> there is only the option "env.java.opts", which can be set in the config. I think you cannot use this to have multiple "-classpath" options when starting Java?
>
> It makes sense to have the option to add something to the classpath. I will check this and fix it. :-) What about "taskmanager.env.java.classpath" and "jobmanager.env.java.classpath" as a configuration key? And maybe an additional "env.java.classpath"?
>
> PS: sorry for the incomplete mail before :D
>
> On 15 Jan 2015, at 01:00, Alexander Alexandrov <[hidden email]> wrote:
>
> > Hi there,
> >
> > is there a canonical / suggested way to customize the classpath of the Flink JM and TM processes?
> >
> > At the moment I hardcoded my way around this by manually changing the following lines like that:
> >
> > # in taskmanager.sh
> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
> >
> > # in jobmanagre.sh
> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
> >
> > Perhaps there is a variable that I can set in conf/flink.yaml that will do the same trick.
>
>
>

[Alexander Alexandrov]

OK, let's be conservative and not blow up the list of configuration values then.

Is there a default point where I should also document these settings?

2015-01-15 15:02 GMT+01:00 Ufuk Celebi <[hidden email]>:

"env.java.classpath" could be used for global class path settings (both JM and TM) and the other ones for JM and TM respectively. I don't know if this is really necessary though. Because no one else asked for it, I think it's fine to go with what suits your use case.

– Ufuk

On 15 Jan 2015, at 14:25, Alexander Alexandrov <[hidden email]> wrote:

> Sorry, I meant between the three. In particular, where will env.java.classpath propagate?
>
> 2015-01-15 14:24 GMT+01:00 Alexander Alexandrov <[hidden email]>:
> No worries, this is the first reply that I get in this thread.
>
> What would be the difference between the two? I can make a PR if we agree on that.
>
> 2015-01-15 11:28 GMT+01:00 Ufuk Celebi <[hidden email]>:
> Hey Alexander,
>
> there is only the option "env.java.opts", which can be set in the config. I think you cannot use this to have multiple "-classpath" options when starting Java?
>
> It makes sense to have the option to add something to the classpath. I will check this and fix it. :-) What about "taskmanager.env.java.classpath" and "jobmanager.env.java.classpath" as a configuration key? And maybe an additional "env.java.classpath"?
>
> PS: sorry for the incomplete mail before :D
>
> On 15 Jan 2015, at 01:00, Alexander Alexandrov <[hidden email]> wrote:
>
> > Hi there,
> >
> > is there a canonical / suggested way to customize the classpath of the Flink JM and TM processes?
> >
> > At the moment I hardcoded my way around this by manually changing the following lines like that:
> >
> > # in taskmanager.sh
> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
> >
> > # in jobmanagre.sh
> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
> >
> > Perhaps there is a variable that I can set in conf/flink.yaml that will do the same trick.
>
>
>

[Ufuk Celebi]

On Thursday, January 15, 2015, Alexander Alexandrov <[hidden email]> wrote:

OK, let's be conservative and not blow up the list of configuration values then. 

Yes :)

 

Is there a default point where I should also document these settings?

In the setup section of the docs. There is an overview page of all config parameters. 

 

2015-01-15 15:02 GMT+01:00 Ufuk Celebi <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;[e-mail]&#39;);" target="_blank">uce@...>:

"env.java.classpath" could be used for global class path settings (both JM and TM) and the other ones for JM and TM respectively. I don't know if this is really necessary though. Because no one else asked for it, I think it's fine to go with what suits your use case.

– Ufuk

On 15 Jan 2015, at 14:25, Alexander Alexandrov <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;[e-mail]&#39;);" target="_blank">alexander.s.alexandrov@...> wrote:

> Sorry, I meant between the three. In particular, where will env.java.classpath propagate?
>
> 2015-01-15 14:24 GMT+01:00 Alexander Alexandrov <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;[e-mail]&#39;);" target="_blank">alexander.s.alexandrov@...>:
> No worries, this is the first reply that I get in this thread.
>
> What would be the difference between the two? I can make a PR if we agree on that.
>
> 2015-01-15 11:28 GMT+01:00 Ufuk Celebi <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;[e-mail]&#39;);" target="_blank">uce@...>:
> Hey Alexander,
>
> there is only the option "env.java.opts", which can be set in the config. I think you cannot use this to have multiple "-classpath" options when starting Java?
>
> It makes sense to have the option to add something to the classpath. I will check this and fix it. :-) What about "taskmanager.env.java.classpath" and "jobmanager.env.java.classpath" as a configuration key? And maybe an additional "env.java.classpath"?
>
> PS: sorry for the incomplete mail before :D
>
> On 15 Jan 2015, at 01:00, Alexander Alexandrov <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;[e-mail]&#39;);" target="_blank">alexander.s.alexandrov@...> wrote:
>
> > Hi there,
> >
> > is there a canonical / suggested way to customize the classpath of the Flink JM and TM processes?
> >
> > At the moment I hardcoded my way around this by manually changing the following lines like that:
> >
> > # in taskmanager.sh
> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_TM_CLASSPATH"
> >
> > # in jobmanagre.sh
> > $JAVA_RUN [...] -classpath "/tmp/classes:$FLINK_JM_CLASSPATH"
> >
> > Perhaps there is a variable that I can set in conf/flink.yaml that will do the same trick.
>
>
>