[0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.

[LINZ, Arnaud]

Hi,
Using last nightly build, it seems that if you call yarn-session.sh with -nm option to give a nice application name, then you cannot submit a job with flink run without specify the ever changing -m <jobManager> address since it does not find it any longer.

Regards,

Arnaud

---

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

[rmetzger0]

Hi Arnaud,

usually, you don't have to manually specify the JobManager address manually with the -m argument, because it is reading it from the conf/.yarn-session.properties file.

Give me a few minutes to reproduce the issue.

On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud <[hidden email]> wrote:

Hi,
Using last nightly build, it seems that if you call yarn-session.sh with -nm option to give a nice application name, then you cannot submit a job with flink run without specify the ever changing -m <jobManager> address since it does not find it any longer.

Regards,

Arnaud

---

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

[LINZ, Arnaud]

Ooops… Seems it was rather a write problem on the conf dir…

Sorry, it works!

 

BTW, it’s not really nice to have an application write the configuration dir ; it’s often a root protected directory in usr/lib/flink. Is there a parameter to put that file elsewhere ?

 

 

De : Robert Metzger [mailto:[hidden email]]
Envoyé : mercredi 26 août 2015 14:42
À : [hidden email]
Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.

 

Hi Arnaud,

 

usually, you don't have to manually specify the JobManager address manually with the -m argument, because it is reading it from the conf/.yarn-session.properties file.

 

Give me a few minutes to reproduce the issue.

 

On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud <[hidden email]> wrote:

Hi,
Using last nightly build, it seems that if you call yarn-session.sh with -nm option to give a nice application name, then you cannot submit a job with flink run without specify the ever changing -m <jobManager> address since it does not find it any longer.

Regards,

Arnaud

 

---

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

 

[rmetzger0]

Great ;)

Not yet, but you are the second user to request this.

I think I'll put the file somewhere else now.

On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud <[hidden email]> wrote:

Ooops… Seems it was rather a write problem on the conf dir…

Sorry, it works!

 

BTW, it’s not really nice to have an application write the configuration dir ; it’s often a root protected directory in usr/lib/flink. Is there a parameter to put that file elsewhere ?

 

 

De : Robert Metzger [mailto:[hidden email]]
Envoyé : mercredi 26 août 2015 14:42
À : [hidden email]
Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.

 

Hi Arnaud,

 

usually, you don't have to manually specify the JobManager address manually with the -m argument, because it is reading it from the conf/.yarn-session.properties file.

 

Give me a few minutes to reproduce the issue.

 

On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud <[hidden email]> wrote:

Hi,
Using last nightly build, it seems that if you call yarn-session.sh with -nm option to give a nice application name, then you cannot submit a job with flink run without specify the ever changing -m <jobManager> address since it does not find it any longer.

Regards,

Arnaud

 

---

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

 

[Aljoscha Krettek]

Hi Arnaud,

I think my answer to Gwenhaël could also be helpful to you:

are you using the one-yarn-cluster-per-job mode of Flink? I.e., you are starting your Flink job with (from the doc):

flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar

If you are, then this is almost possible on the current version of Flink. What you have to do is copy the conf directory of Flink to a separate directory that is specific to your job. There you make your modifications to the log configuration etc. Then, when you start your job you do this instead:

export FLINK_CONF_DIR=/path/to/my/conf

flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar

You can easily put this into your startup script, of course.

I said almost possible because this requires a small fix in bin/flink. Around line 130 this line:

FLINK_CONF_DIR=$FLINK_ROOT_DIR_MANGLED/conf

needs to be replaced by this line:

if [ -z "$FLINK_CONF_DIR" ]; then FLINK_CONF_DIR=$FLINK_ROOT_DIR_MANGLED/conf; fi

(We will fix this in the upcoming version and the 0.9.1 bugfix release.)

Does this help? Let us know if you are not using the one-yarn-cluster-per-job mode, then we'll have to try to find another solution.

Forget the stuff about one-yarn-cluster-per-job but the other stuff should be right what you want. I hope it helps.

Best,

Aljoscha

On Wed, 26 Aug 2015 at 15:19 LINZ, Arnaud <[hidden email]> wrote:

Ooops… Seems it was rather a write problem on the conf dir…

Sorry, it works!

 

BTW, it’s not really nice to have an application write the configuration dir ; it’s often a root protected directory in usr/lib/flink. Is there a parameter to put that file elsewhere ?

 

 

De : Robert Metzger [mailto:[hidden email]]
Envoyé : mercredi 26 août 2015 14:42
À : [hidden email]
Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.

 

Hi Arnaud,

 

usually, you don't have to manually specify the JobManager address manually with the -m argument, because it is reading it from the conf/.yarn-session.properties file.

 

Give me a few minutes to reproduce the issue.

 

On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud <[hidden email]> wrote:

Hi,
Using last nightly build, it seems that if you call yarn-session.sh with -nm option to give a nice application name, then you cannot submit a job with flink run without specify the ever changing -m <jobManager> address since it does not find it any longer.

Regards,

Arnaud

 

---

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

 

[Maximilian Michels]

Can't we write the file to the system's temp directory or the user
home? IMHO this is more standard practice for these type of session
information.

On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger <[hidden email]> wrote:

> Great ;)
>
> Not yet, but you are the second user to request this.
> I think I'll put the file somewhere else now.
>
> On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud <[hidden email]>
> wrote:
>>
>> Ooops… Seems it was rather a write problem on the conf dir…
>>
>> Sorry, it works!
>>
>>
>>
>> BTW, it’s not really nice to have an application write the configuration
>> dir ; it’s often a root protected directory in usr/lib/flink. Is there a
>> parameter to put that file elsewhere ?
>>
>>
>>
>>
>>
>> De : Robert Metzger [mailto:[hidden email]]
>> Envoyé : mercredi 26 août 2015 14:42
>> À : [hidden email]
>> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session
>> -nm), flink run without -m fails.
>>
>>
>>
>> Hi Arnaud,
>>
>>
>>
>> usually, you don't have to manually specify the JobManager address
>> manually with the -m argument, because it is reading it from the
>> conf/.yarn-session.properties file.
>>
>>
>>
>> Give me a few minutes to reproduce the issue.
>>
>>
>>
>> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud <[hidden email]>
>> wrote:
>>
>> Hi,
>> Using last nightly build, it seems that if you call yarn-session.sh with
>> -nm option to give a nice application name, then you cannot submit a job
>> with flink run without specify the ever changing -m <jobManager> address
>> since it does not find it any longer.
>>
>> Regards,
>>
>> Arnaud
>>
>>
>>
>> ________________________________
>>
>>
>> L'intégrité de ce message n'étant pas assurée sur internet, la société
>> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
>> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous
>> n'êtes pas destinataire de ce message, merci de le détruire et d'avertir
>> l'expéditeur.
>>
>> The integrity of this message cannot be guaranteed on the Internet. The
>> company that sent this message cannot therefore be held liable for its
>> content nor attachments. Any unauthorized use or dissemination is
>> prohibited. If you are not the intended recipient of this message, then
>> please delete it and notify the sender.
>>
>>
>
>

[rmetzger0]

Yep. I think the start-*.sh scripts are also writing the PID to tmp.

On Wed, Aug 26, 2015 at 3:30 PM, Maximilian Michels <[hidden email]> wrote:

Can't we write the file to the system's temp directory or the user
home? IMHO this is more standard practice for these type of session
information.

On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger <[hidden email]> wrote:
> Great ;)
>
> Not yet, but you are the second user to request this.
> I think I'll put the file somewhere else now.
>
> On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud <[hidden email]>
> wrote:
>>
>> Ooops… Seems it was rather a write problem on the conf dir…
>>
>> Sorry, it works!
>>
>>
>>
>> BTW, it’s not really nice to have an application write the configuration
>> dir ; it’s often a root protected directory in usr/lib/flink. Is there a
>> parameter to put that file elsewhere ?
>>
>>
>>
>>
>>
>> De : Robert Metzger [mailto:[hidden email]]
>> Envoyé : mercredi 26 août 2015 14:42
>> À : [hidden email]
>> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session
>> -nm), flink run without -m fails.
>>
>>
>>
>> Hi Arnaud,
>>
>>
>>
>> usually, you don't have to manually specify the JobManager address
>> manually with the -m argument, because it is reading it from the
>> conf/.yarn-session.properties file.
>>
>>
>>
>> Give me a few minutes to reproduce the issue.
>>
>>
>>
>> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud <[hidden email]>
>> wrote:
>>
>> Hi,
>> Using last nightly build, it seems that if you call yarn-session.sh with
>> -nm option to give a nice application name, then you cannot submit a job
>> with flink run without specify the ever changing -m <jobManager> address
>> since it does not find it any longer.
>>
>> Regards,
>>
>> Arnaud
>>
>>
>>
>> ________________________________
>>
>>
>> L'intégrité de ce message n'étant pas assurée sur internet, la société
>> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
>> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous
>> n'êtes pas destinataire de ce message, merci de le détruire et d'avertir
>> l'expéditeur.
>>
>> The integrity of this message cannot be guaranteed on the Internet. The
>> company that sent this message cannot therefore be held liable for its
>> content nor attachments. Any unauthorized use or dissemination is
>> prohibited. If you are not the intended recipient of this message, then
>> please delete it and notify the sender.
>>
>>
>
>

[Maximilian Michels]

The only problem with writing the temp is that it will be gone after a
restart. While this is not important for PIDs because the system has
been restarted anyways, this can actually be a problem if you want to
resume a YARN cluster after you have restarted your system.

On Wed, Aug 26, 2015 at 3:34 PM, Robert Metzger <[hidden email]> wrote:

> Yep. I think the start-*.sh scripts are also writing the PID to tmp.
>
> On Wed, Aug 26, 2015 at 3:30 PM, Maximilian Michels <[hidden email]> wrote:
>>
>> Can't we write the file to the system's temp directory or the user
>> home? IMHO this is more standard practice for these type of session
>> information.
>>
>> On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger <[hidden email]>
>> wrote:
>> > Great ;)
>> >
>> > Not yet, but you are the second user to request this.
>> > I think I'll put the file somewhere else now.
>> >
>> > On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud <[hidden email]>
>> > wrote:
>> >>
>> >> Ooops… Seems it was rather a write problem on the conf dir…
>> >>
>> >> Sorry, it works!
>> >>
>> >>
>> >>
>> >> BTW, it’s not really nice to have an application write the
>> >> configuration
>> >> dir ; it’s often a root protected directory in usr/lib/flink. Is there
>> >> a
>> >> parameter to put that file elsewhere ?
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> De : Robert Metzger [mailto:[hidden email]]
>> >> Envoyé : mercredi 26 août 2015 14:42
>> >> À : [hidden email]
>> >> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session
>> >> -nm), flink run without -m fails.
>> >>
>> >>
>> >>
>> >> Hi Arnaud,
>> >>
>> >>
>> >>
>> >> usually, you don't have to manually specify the JobManager address
>> >> manually with the -m argument, because it is reading it from the
>> >> conf/.yarn-session.properties file.
>> >>
>> >>
>> >>
>> >> Give me a few minutes to reproduce the issue.
>> >>
>> >>
>> >>
>> >> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud
>> >> <[hidden email]>
>> >> wrote:
>> >>
>> >> Hi,
>> >> Using last nightly build, it seems that if you call yarn-session.sh
>> >> with
>> >> -nm option to give a nice application name, then you cannot submit a
>> >> job
>> >> with flink run without specify the ever changing -m <jobManager>
>> >> address
>> >> since it does not find it any longer.
>> >>
>> >> Regards,
>> >>
>> >> Arnaud
>> >>
>> >>
>> >>
>> >> ________________________________
>> >>
>> >>
>> >> L'intégrité de ce message n'étant pas assurée sur internet, la société
>> >> expéditrice ne peut être tenue responsable de son contenu ni de ses
>> >> pièces
>> >> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
>> >> vous
>> >> n'êtes pas destinataire de ce message, merci de le détruire et
>> >> d'avertir
>> >> l'expéditeur.
>> >>
>> >> The integrity of this message cannot be guaranteed on the Internet. The
>> >> company that sent this message cannot therefore be held liable for its
>> >> content nor attachments. Any unauthorized use or dissemination is
>> >> prohibited. If you are not the intended recipient of this message, then
>> >> please delete it and notify the sender.
>> >>
>> >>
>> >
>> >
>
>

[rmetzger0]

Therefore, my change will include a configuration option to set a custom location for the file.

On Wed, Aug 26, 2015 at 5:55 PM, Maximilian Michels <[hidden email]> wrote:

The only problem with writing the temp is that it will be gone after a
restart. While this is not important for PIDs because the system has
been restarted anyways, this can actually be a problem if you want to
resume a YARN cluster after you have restarted your system.

On Wed, Aug 26, 2015 at 3:34 PM, Robert Metzger <[hidden email]> wrote:
> Yep. I think the start-*.sh scripts are also writing the PID to tmp.
>
> On Wed, Aug 26, 2015 at 3:30 PM, Maximilian Michels <[hidden email]> wrote:
>>
>> Can't we write the file to the system's temp directory or the user
>> home? IMHO this is more standard practice for these type of session
>> information.
>>
>> On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger <[hidden email]>
>> wrote:
>> > Great ;)
>> >
>> > Not yet, but you are the second user to request this.
>> > I think I'll put the file somewhere else now.
>> >
>> > On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud <[hidden email]>
>> > wrote:
>> >>
>> >> Ooops… Seems it was rather a write problem on the conf dir…
>> >>
>> >> Sorry, it works!
>> >>
>> >>
>> >>
>> >> BTW, it’s not really nice to have an application write the
>> >> configuration
>> >> dir ; it’s often a root protected directory in usr/lib/flink. Is there
>> >> a
>> >> parameter to put that file elsewhere ?
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> De : Robert Metzger [mailto:[hidden email]]
>> >> Envoyé : mercredi 26 août 2015 14:42
>> >> À : [hidden email]
>> >> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session
>> >> -nm), flink run without -m fails.
>> >>
>> >>
>> >>
>> >> Hi Arnaud,
>> >>
>> >>
>> >>
>> >> usually, you don't have to manually specify the JobManager address
>> >> manually with the -m argument, because it is reading it from the
>> >> conf/.yarn-session.properties file.
>> >>
>> >>
>> >>
>> >> Give me a few minutes to reproduce the issue.
>> >>
>> >>
>> >>
>> >> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud
>> >> <[hidden email]>
>> >> wrote:
>> >>
>> >> Hi,
>> >> Using last nightly build, it seems that if you call yarn-session.sh
>> >> with
>> >> -nm option to give a nice application name, then you cannot submit a
>> >> job
>> >> with flink run without specify the ever changing -m <jobManager>
>> >> address
>> >> since it does not find it any longer.
>> >>
>> >> Regards,
>> >>
>> >> Arnaud
>> >>
>> >>
>> >>
>> >> ________________________________
>> >>
>> >>
>> >> L'intégrité de ce message n'étant pas assurée sur internet, la société
>> >> expéditrice ne peut être tenue responsable de son contenu ni de ses
>> >> pièces
>> >> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
>> >> vous
>> >> n'êtes pas destinataire de ce message, merci de le détruire et
>> >> d'avertir
>> >> l'expéditeur.
>> >>
>> >> The integrity of this message cannot be guaranteed on the Internet. The
>> >> company that sent this message cannot therefore be held liable for its
>> >> content nor attachments. Any unauthorized use or dissemination is
>> >> prohibited. If you are not the intended recipient of this message, then
>> >> please delete it and notify the sender.
>> >>
>> >>
>> >
>> >
>
>

[Maximilian Michels]

Nice. More configuration options :)

On Wed, Aug 26, 2015 at 5:58 PM, Robert Metzger <[hidden email]> wrote:

> Therefore, my change will include a configuration option to set a custom
> location for the file.
>
> On Wed, Aug 26, 2015 at 5:55 PM, Maximilian Michels <[hidden email]> wrote:
>>
>> The only problem with writing the temp is that it will be gone after a
>> restart. While this is not important for PIDs because the system has
>> been restarted anyways, this can actually be a problem if you want to
>> resume a YARN cluster after you have restarted your system.
>>
>> On Wed, Aug 26, 2015 at 3:34 PM, Robert Metzger <[hidden email]>
>> wrote:
>> > Yep. I think the start-*.sh scripts are also writing the PID to tmp.
>> >
>> > On Wed, Aug 26, 2015 at 3:30 PM, Maximilian Michels <[hidden email]>
>> > wrote:
>> >>
>> >> Can't we write the file to the system's temp directory or the user
>> >> home? IMHO this is more standard practice for these type of session
>> >> information.
>> >>
>> >> On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger <[hidden email]>
>> >> wrote:
>> >> > Great ;)
>> >> >
>> >> > Not yet, but you are the second user to request this.
>> >> > I think I'll put the file somewhere else now.
>> >> >
>> >> > On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud
>> >> > <[hidden email]>
>> >> > wrote:
>> >> >>
>> >> >> Ooops… Seems it was rather a write problem on the conf dir…
>> >> >>
>> >> >> Sorry, it works!
>> >> >>
>> >> >>
>> >> >>
>> >> >> BTW, it’s not really nice to have an application write the
>> >> >> configuration
>> >> >> dir ; it’s often a root protected directory in usr/lib/flink. Is
>> >> >> there
>> >> >> a
>> >> >> parameter to put that file elsewhere ?
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> De : Robert Metzger [mailto:[hidden email]]
>> >> >> Envoyé : mercredi 26 août 2015 14:42
>> >> >> À : [hidden email]
>> >> >> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application
>> >> >> (yarn-session
>> >> >> -nm), flink run without -m fails.
>> >> >>
>> >> >>
>> >> >>
>> >> >> Hi Arnaud,
>> >> >>
>> >> >>
>> >> >>
>> >> >> usually, you don't have to manually specify the JobManager address
>> >> >> manually with the -m argument, because it is reading it from the
>> >> >> conf/.yarn-session.properties file.
>> >> >>
>> >> >>
>> >> >>
>> >> >> Give me a few minutes to reproduce the issue.
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud
>> >> >> <[hidden email]>
>> >> >> wrote:
>> >> >>
>> >> >> Hi,
>> >> >> Using last nightly build, it seems that if you call yarn-session.sh
>> >> >> with
>> >> >> -nm option to give a nice application name, then you cannot submit a
>> >> >> job
>> >> >> with flink run without specify the ever changing -m <jobManager>
>> >> >> address
>> >> >> since it does not find it any longer.
>> >> >>
>> >> >> Regards,
>> >> >>
>> >> >> Arnaud
>> >> >>
>> >> >>
>> >> >>
>> >> >> ________________________________
>> >> >>
>> >> >>
>> >> >> L'intégrité de ce message n'étant pas assurée sur internet, la
>> >> >> société
>> >> >> expéditrice ne peut être tenue responsable de son contenu ni de ses
>> >> >> pièces
>> >> >> jointes. Toute utilisation ou diffusion non autorisée est interdite.
>> >> >> Si
>> >> >> vous
>> >> >> n'êtes pas destinataire de ce message, merci de le détruire et
>> >> >> d'avertir
>> >> >> l'expéditeur.
>> >> >>
>> >> >> The integrity of this message cannot be guaranteed on the Internet.
>> >> >> The
>> >> >> company that sent this message cannot therefore be held liable for
>> >> >> its
>> >> >> content nor attachments. Any unauthorized use or dissemination is
>> >> >> prohibited. If you are not the intended recipient of this message,
>> >> >> then
>> >> >> please delete it and notify the sender.
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >
>> >
>
>

[rmetzger0]

Just as a quick update on this: The change has been merged into 0.10-SNAPSHOT.

Flink is now writing the jobmanager connection information into the temp directory.

On Wed, Aug 26, 2015 at 6:00 PM, Maximilian Michels <[hidden email]> wrote:

Nice. More configuration options :)

On Wed, Aug 26, 2015 at 5:58 PM, Robert Metzger <[hidden email]> wrote:
> Therefore, my change will include a configuration option to set a custom
> location for the file.
>
> On Wed, Aug 26, 2015 at 5:55 PM, Maximilian Michels <[hidden email]> wrote:
>>
>> The only problem with writing the temp is that it will be gone after a
>> restart. While this is not important for PIDs because the system has
>> been restarted anyways, this can actually be a problem if you want to
>> resume a YARN cluster after you have restarted your system.
>>
>> On Wed, Aug 26, 2015 at 3:34 PM, Robert Metzger <[hidden email]>
>> wrote:
>> > Yep. I think the start-*.sh scripts are also writing the PID to tmp.
>> >
>> > On Wed, Aug 26, 2015 at 3:30 PM, Maximilian Michels <[hidden email]>
>> > wrote:
>> >>
>> >> Can't we write the file to the system's temp directory or the user
>> >> home? IMHO this is more standard practice for these type of session
>> >> information.
>> >>
>> >> On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger <[hidden email]>
>> >> wrote:
>> >> > Great ;)
>> >> >
>> >> > Not yet, but you are the second user to request this.
>> >> > I think I'll put the file somewhere else now.
>> >> >
>> >> > On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud
>> >> > <[hidden email]>
>> >> > wrote:
>> >> >>
>> >> >> Ooops… Seems it was rather a write problem on the conf dir…
>> >> >>
>> >> >> Sorry, it works!
>> >> >>
>> >> >>
>> >> >>
>> >> >> BTW, it’s not really nice to have an application write the
>> >> >> configuration
>> >> >> dir ; it’s often a root protected directory in usr/lib/flink. Is
>> >> >> there
>> >> >> a
>> >> >> parameter to put that file elsewhere ?
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> De : Robert Metzger [mailto:[hidden email]]
>> >> >> Envoyé : mercredi 26 août 2015 14:42
>> >> >> À : [hidden email]
>> >> >> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application
>> >> >> (yarn-session
>> >> >> -nm), flink run without -m fails.
>> >> >>
>> >> >>
>> >> >>
>> >> >> Hi Arnaud,
>> >> >>
>> >> >>
>> >> >>
>> >> >> usually, you don't have to manually specify the JobManager address
>> >> >> manually with the -m argument, because it is reading it from the
>> >> >> conf/.yarn-session.properties file.
>> >> >>
>> >> >>
>> >> >>
>> >> >> Give me a few minutes to reproduce the issue.
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud
>> >> >> <[hidden email]>
>> >> >> wrote:
>> >> >>
>> >> >> Hi,
>> >> >> Using last nightly build, it seems that if you call yarn-session.sh
>> >> >> with
>> >> >> -nm option to give a nice application name, then you cannot submit a
>> >> >> job
>> >> >> with flink run without specify the ever changing -m <jobManager>
>> >> >> address
>> >> >> since it does not find it any longer.
>> >> >>
>> >> >> Regards,
>> >> >>
>> >> >> Arnaud
>> >> >>
>> >> >>
>> >> >>
>> >> >> ________________________________
>> >> >>
>> >> >>
>> >> >> L'intégrité de ce message n'étant pas assurée sur internet, la
>> >> >> société
>> >> >> expéditrice ne peut être tenue responsable de son contenu ni de ses
>> >> >> pièces
>> >> >> jointes. Toute utilisation ou diffusion non autorisée est interdite.
>> >> >> Si
>> >> >> vous
>> >> >> n'êtes pas destinataire de ce message, merci de le détruire et
>> >> >> d'avertir
>> >> >> l'expéditeur.
>> >> >>
>> >> >> The integrity of this message cannot be guaranteed on the Internet.
>> >> >> The
>> >> >> company that sent this message cannot therefore be held liable for
>> >> >> its
>> >> >> content nor attachments. Any unauthorized use or dissemination is
>> >> >> prohibited. If you are not the intended recipient of this message,
>> >> >> then
>> >> >> please delete it and notify the sender.
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >
>> >
>
>