(DEPRECATED) Apache Flink User Mailing List archive.

How to run a job with job cluster mode on top of mesos?

Classic

List

Threaded

8 messages Options

Jacky Yin 殷传旺

How to run a job with job cluster mode on top of mesos?

Anybody can help? I found the help documentation about how to do with the job cluster mode for yarn, docker and k8s. However only the help of session cluster mode is provided in flink web site for mesos.

It looks like the shell “mesos-appmaster-job.sh” should be the right one to run a job with the mode of job cluster on top of mesos. However I cannot find any help or example for this shell.

Any help will be greatly appreciated.

Thanks!

Jacky Yin

发件人: Jacky Yin 殷传旺 <[hidden email]>
日期: 2019年3月21日星期四下午2:31
收件人: "[hidden email]" <[hidden email]>
主题: Documentation of mesos-appmaster-job.sh

Hello All,

I cannot find any documentation or help about how to use $flin_home/bin/mesos-appmaster-job.sh. Anybody help?

Thanks!

Jacky Yin

Till Rohrmann

Re: How to run a job with job cluster mode on top of mesos?

Hi Jacky,

you're right that we are currently lacking documentation for the `mesos-appmaster-job.sh` script. I've added a JIRA issue to cover this [1].

In order to use this script you first need to store a serialized version of the `JobGraph` you want to run somewhere where the Mesos appmaster can read it. Moreover, you need to make sure that the user code jars are on the classpath (e.g. putting them in the lib directory). See how the AbstractYarnClusterDescriptor serializes the `JobGraph` [2] for some details. The last thing before you can start the MesosJobClusterEntrypoint is to specify the path to the serialized job graph via the `internal.jobgraph-path` configuration option which you can specify in your flink-conf.yaml or pass in via dynamic property.

I know that this is not super streamlined and needs to be improved. E.g. one could do it similarly to the `StandaloneJobClusterEntrypoint` that one includes the user code jar and specifies the class name of the user code to load. That way one would not need to generate the JobGraph yourself and then serialize it.

I hope that I could help you a little bit.

[1] https://issues.apache.org/jira/browse/FLINK-12020

[2] https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L846

Cheers,

Till

On Fri, Mar 22, 2019 at 7:24 AM Jacky Yin 殷传旺 <[hidden email]> wrote:

Anybody can help? I found the help documentation about how to do with the job cluster mode for yarn, docker and k8s. However only the help of session cluster mode is provided in flink web site for mesos.

It looks like the shell “mesos-appmaster-job.sh” should be the right one to run a job with the mode of job cluster on top of mesos. However I cannot find any help or example for this shell.

Any help will be greatly appreciated.

Thanks!

Jacky Yin

发件人: Jacky Yin 殷传旺 <[hidden email]>
日期: 2019年3月21日星期四下午2:31
收件人: "[hidden email]" <[hidden email]>
主题: Documentation of mesos-appmaster-job.sh

Hello All,

I cannot find any documentation or help about how to use $flin_home/bin/mesos-appmaster-job.sh. Anybody help?

Thanks!

Jacky Yin

Jacky Yin 殷传旺

Re: How to run a job with job cluster mode on top of mesos?

Hello Till,

Thanks very much for the clear steps. And I noticed that you have already created a jira ticket for this issue and I would like to contribute it. (I have already assigned it to me). 😊

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年3月26日星期二下午6:31
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

Hi Jacky,

you're right that we are currently lacking documentation for the `mesos-appmaster-job.sh` script. I've added a JIRA issue to cover this [1].

I hope that I could help you a little bit.

[1] https://issues.apache.org/jira/browse/FLINK-12020

[2] https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L846

Cheers,

Till

On Fri, Mar 22, 2019 at 7:24 AM Jacky Yin 殷传旺 <[hidden email]> wrote:

Anybody can help? I found the help documentation about how to do with the job cluster mode for yarn, docker and k8s. However only the help of session cluster mode is provided in flink web site for mesos.

It looks like the shell “mesos-appmaster-job.sh” should be the right one to run a job with the mode of job cluster on top of mesos. However I cannot find any help or example for this shell.

Any help will be greatly appreciated.

Thanks!

Jacky Yin

发件人: Jacky Yin 殷传旺 <[hidden email]>
日期: 2019年3月21日星期四下午2:31
收件人: "[hidden email]" <[hidden email]>
主题: Documentation of mesos-appmaster-job.sh

Hello All,

I cannot find any documentation or help about how to use $flin_home/bin/mesos-appmaster-job.sh. Anybody help?

Thanks!

Jacky Yin

Till Rohrmann

Re: How to run a job with job cluster mode on top of mesos?

Thanks a lot Jacky.

Cheers,

Till

On Fri, Mar 29, 2019 at 1:50 PM Jacky Yin 殷传旺 <[hidden email]> wrote:

Hello Till,

Thanks very much for the clear steps. And I noticed that you have already created a jira ticket for this issue and I would like to contribute it. (I have already assigned it to me). 😊

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年3月26日星期二下午6:31
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

Hi Jacky,

you're right that we are currently lacking documentation for the `mesos-appmaster-job.sh` script. I've added a JIRA issue to cover this [1].

In order to use this script you first need to store a serialized version of the `JobGraph` you want to run somewhere where the Mesos appmaster can read it. Moreover, you need to make sure that the user code jars are on the classpath (e.g. putting them in the lib directory). See how the AbstractYarnClusterDescriptor serializes the `JobGraph` [2] for some details. The last thing before you can start the MesosJobClusterEntrypoint is to specify the path to the serialized job graph via the `internal.jobgraph-path` configuration option which you can specify in your flink-conf.yaml or pass in via dynamic property.

I know that this is not super streamlined and needs to be improved. E.g. one could do it similarly to the `StandaloneJobClusterEntrypoint` that one includes the user code jar and specifies the class name of the user code to load. That way one would not need to generate the JobGraph yourself and then serialize it.

I hope that I could help you a little bit.

[1] https://issues.apache.org/jira/browse/FLINK-12020

[2] https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L846

Cheers,

Till

On Fri, Mar 22, 2019 at 7:24 AM Jacky Yin 殷传旺 <[hidden email]> wrote:

Anybody can help? I found the help documentation about how to do with the job cluster mode for yarn, docker and k8s. However only the help of session cluster mode is provided in flink web site for mesos.

It looks like the shell “mesos-appmaster-job.sh” should be the right one to run a job with the mode of job cluster on top of mesos. However I cannot find any help or example for this shell.

Any help will be greatly appreciated.

Thanks!

Jacky Yin

发件人: Jacky Yin 殷传旺 <[hidden email]>
日期: 2019年3月21日星期四下午2:31
收件人: "[hidden email]" <[hidden email]>
主题: Documentation of mesos-appmaster-job.sh

Hello All,

I cannot find any documentation or help about how to use $flin_home/bin/mesos-appmaster-job.sh. Anybody help?

Thanks!

Jacky Yin

Jacky Yin 殷传旺

Re: How to run a job with job cluster mode on top of mesos?

Hello Till,

I submitted a PR(#8084) for this issue. Could you help review it?

Many thanks!

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年3月29日星期五下午11:06
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

Thanks a lot Jacky.

Cheers,

Till

On Fri, Mar 29, 2019 at 1:50 PM Jacky Yin 殷传旺 <[hidden email]> wrote:

Hello Till,

Thanks very much for the clear steps. And I noticed that you have already created a jira ticket for this issue and I would like to contribute it. (I have already assigned it to me). 😊

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年3月26日星期二下午6:31
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

Hi Jacky,

you're right that we are currently lacking documentation for the `mesos-appmaster-job.sh` script. I've added a JIRA issue to cover this [1].

In order to use this script you first need to store a serialized version of the `JobGraph` you want to run somewhere where the Mesos appmaster can read it. Moreover, you need to make sure that the user code jars are on the classpath (e.g. putting them in the lib directory). See how the AbstractYarnClusterDescriptor serializes the `JobGraph` [2] for some details. The last thing before you can start the MesosJobClusterEntrypoint is to specify the path to the serialized job graph via the `internal.jobgraph-path` configuration option which you can specify in your flink-conf.yaml or pass in via dynamic property.

I know that this is not super streamlined and needs to be improved. E.g. one could do it similarly to the `StandaloneJobClusterEntrypoint` that one includes the user code jar and specifies the class name of the user code to load. That way one would not need to generate the JobGraph yourself and then serialize it.

I hope that I could help you a little bit.

[1] https://issues.apache.org/jira/browse/FLINK-12020

[2] https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L846

Cheers,

Till

On Fri, Mar 22, 2019 at 7:24 AM Jacky Yin 殷传旺 <[hidden email]> wrote:

Anybody can help? I found the help documentation about how to do with the job cluster mode for yarn, docker and k8s. However only the help of session cluster mode is provided in flink web site for mesos.

It looks like the shell “mesos-appmaster-job.sh” should be the right one to run a job with the mode of job cluster on top of mesos. However I cannot find any help or example for this shell.

Any help will be greatly appreciated.

Thanks!

Jacky Yin

发件人: Jacky Yin 殷传旺 <[hidden email]>
日期: 2019年3月21日星期四下午2:31
收件人: "[hidden email]" <[hidden email]>
主题: Documentation of mesos-appmaster-job.sh

Hello All,

I cannot find any documentation or help about how to use $flin_home/bin/mesos-appmaster-job.sh. Anybody help?

Thanks!

Jacky Yin

Till Rohrmann

Re: How to run a job with job cluster mode on top of mesos?

Sure I will help with the review. Thanks for opening the PR Jacky.

Cheers,

Till

On Tue, Apr 2, 2019 at 2:17 AM Jacky Yin 殷传旺 <[hidden email]> wrote:

Hello Till,

I submitted a PR(#8084) for this issue. Could you help review it?

Many thanks!

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年3月29日星期五下午11:06
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

Thanks a lot Jacky.

Cheers,

Till

On Fri, Mar 29, 2019 at 1:50 PM Jacky Yin 殷传旺 <[hidden email]> wrote:

Hello Till,

Thanks very much for the clear steps. And I noticed that you have already created a jira ticket for this issue and I would like to contribute it. (I have already assigned it to me). 😊

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年3月26日星期二下午6:31
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

Hi Jacky,

you're right that we are currently lacking documentation for the `mesos-appmaster-job.sh` script. I've added a JIRA issue to cover this [1].

In order to use this script you first need to store a serialized version of the `JobGraph` you want to run somewhere where the Mesos appmaster can read it. Moreover, you need to make sure that the user code jars are on the classpath (e.g. putting them in the lib directory). See how the AbstractYarnClusterDescriptor serializes the `JobGraph` [2] for some details. The last thing before you can start the MesosJobClusterEntrypoint is to specify the path to the serialized job graph via the `internal.jobgraph-path` configuration option which you can specify in your flink-conf.yaml or pass in via dynamic property.

I know that this is not super streamlined and needs to be improved. E.g. one could do it similarly to the `StandaloneJobClusterEntrypoint` that one includes the user code jar and specifies the class name of the user code to load. That way one would not need to generate the JobGraph yourself and then serialize it.

I hope that I could help you a little bit.

[1] https://issues.apache.org/jira/browse/FLINK-12020

[2] https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L846

Cheers,

Till

On Fri, Mar 22, 2019 at 7:24 AM Jacky Yin 殷传旺 <[hidden email]> wrote:

Anybody can help? I found the help documentation about how to do with the job cluster mode for yarn, docker and k8s. However only the help of session cluster mode is provided in flink web site for mesos.

It looks like the shell “mesos-appmaster-job.sh” should be the right one to run a job with the mode of job cluster on top of mesos. However I cannot find any help or example for this shell.

Any help will be greatly appreciated.

Thanks!

Jacky Yin

发件人: Jacky Yin 殷传旺 <[hidden email]>
日期: 2019年3月21日星期四下午2:31
收件人: "[hidden email]" <[hidden email]>
主题: Documentation of mesos-appmaster-job.sh

Hello All,

I cannot find any documentation or help about how to use $flin_home/bin/mesos-appmaster-job.sh. Anybody help?

Thanks!

Jacky Yin

Till Rohrmann

Re: How to run a job with job cluster mode on top of mesos?

By the way, did the Mesos job mode work for you in the end?

On Tue, Apr 2, 2019 at 7:47 AM Till Rohrmann <[hidden email]> wrote:

Sure I will help with the review. Thanks for opening the PR Jacky.

Cheers,
Till

On Tue, Apr 2, 2019 at 2:17 AM Jacky Yin 殷传旺 <[hidden email]> wrote:

Hello Till,

I submitted a PR(#8084) for this issue. Could you help review it?

Many thanks!

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年3月29日星期五下午11:06
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

Thanks a lot Jacky.

Cheers,

Till

On Fri, Mar 29, 2019 at 1:50 PM Jacky Yin 殷传旺 <[hidden email]> wrote:

Hello Till,

Thanks very much for the clear steps. And I noticed that you have already created a jira ticket for this issue and I would like to contribute it. (I have already assigned it to me). 😊

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年3月26日星期二下午6:31
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

Hi Jacky,

you're right that we are currently lacking documentation for the `mesos-appmaster-job.sh` script. I've added a JIRA issue to cover this [1].

In order to use this script you first need to store a serialized version of the `JobGraph` you want to run somewhere where the Mesos appmaster can read it. Moreover, you need to make sure that the user code jars are on the classpath (e.g. putting them in the lib directory). See how the AbstractYarnClusterDescriptor serializes the `JobGraph` [2] for some details. The last thing before you can start the MesosJobClusterEntrypoint is to specify the path to the serialized job graph via the `internal.jobgraph-path` configuration option which you can specify in your flink-conf.yaml or pass in via dynamic property.

I know that this is not super streamlined and needs to be improved. E.g. one could do it similarly to the `StandaloneJobClusterEntrypoint` that one includes the user code jar and specifies the class name of the user code to load. That way one would not need to generate the JobGraph yourself and then serialize it.

I hope that I could help you a little bit.

[1] https://issues.apache.org/jira/browse/FLINK-12020

[2] https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L846

Cheers,

Till

On Fri, Mar 22, 2019 at 7:24 AM Jacky Yin 殷传旺 <[hidden email]> wrote:

Anybody can help? I found the help documentation about how to do with the job cluster mode for yarn, docker and k8s. However only the help of session cluster mode is provided in flink web site for mesos.

It looks like the shell “mesos-appmaster-job.sh” should be the right one to run a job with the mode of job cluster on top of mesos. However I cannot find any help or example for this shell.

Any help will be greatly appreciated.

Thanks!

Jacky Yin

发件人: Jacky Yin 殷传旺 <[hidden email]>
日期: 2019年3月21日星期四下午2:31
收件人: "[hidden email]" <[hidden email]>
主题: Documentation of mesos-appmaster-job.sh

Hello All,

I cannot find any documentation or help about how to use $flin_home/bin/mesos-appmaster-job.sh. Anybody help?

Thanks!

Jacky Yin

Jacky Yin 殷传旺

Re: How to run a job with job cluster mode on top of mesos?

Yes, it worked for me. However, just like what you said, it is not that straightforward, so I would like to learn from ` StandaloneJobClusterEntrypoint ` and try to enhance the ` MesosJobClusterEntrypoint`. 😊

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年4月2日星期二下午10:50
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

By the way, did the Mesos job mode work for you in the end?

On Tue, Apr 2, 2019 at 7:47 AM Till Rohrmann <[hidden email]> wrote:

Sure I will help with the review. Thanks for opening the PR Jacky.

Cheers,

Till

On Tue, Apr 2, 2019 at 2:17 AM Jacky Yin 殷传旺 <[hidden email]> wrote:

Hello Till,

I submitted a PR(#8084) for this issue. Could you help review it?

Many thanks!

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年3月29日星期五下午11:06
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

Thanks a lot Jacky.

Cheers,

Till

On Fri, Mar 29, 2019 at 1:50 PM Jacky Yin 殷传旺 <[hidden email]> wrote:

Hello Till,

Thanks very much for the clear steps. And I noticed that you have already created a jira ticket for this issue and I would like to contribute it. (I have already assigned it to me). 😊

Jacky Yin

发件人: Till Rohrmann <[hidden email]>
日期: 2019年3月26日星期二下午6:31
收件人: Jacky Yin 殷传旺 <[hidden email]>
抄送: "[hidden email]" <[hidden email]>
主题: Re: How to run a job with job cluster mode on top of mesos?

Hi Jacky,

you're right that we are currently lacking documentation for the `mesos-appmaster-job.sh` script. I've added a JIRA issue to cover this [1].

In order to use this script you first need to store a serialized version of the `JobGraph` you want to run somewhere where the Mesos appmaster can read it. Moreover, you need to make sure that the user code jars are on the classpath (e.g. putting them in the lib directory). See how the AbstractYarnClusterDescriptor serializes the `JobGraph` [2] for some details. The last thing before you can start the MesosJobClusterEntrypoint is to specify the path to the serialized job graph via the `internal.jobgraph-path` configuration option which you can specify in your flink-conf.yaml or pass in via dynamic property.

I know that this is not super streamlined and needs to be improved. E.g. one could do it similarly to the `StandaloneJobClusterEntrypoint` that one includes the user code jar and specifies the class name of the user code to load. That way one would not need to generate the JobGraph yourself and then serialize it.

I hope that I could help you a little bit.

[1] https://issues.apache.org/jira/browse/FLINK-12020

[2] https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L846

Cheers,

Till

On Fri, Mar 22, 2019 at 7:24 AM Jacky Yin 殷传旺 <[hidden email]> wrote:

Anybody can help? I found the help documentation about how to do with the job cluster mode for yarn, docker and k8s. However only the help of session cluster mode is provided in flink web site for mesos.

It looks like the shell “mesos-appmaster-job.sh” should be the right one to run a job with the mode of job cluster on top of mesos. However I cannot find any help or example for this shell.

Any help will be greatly appreciated.

Thanks!

Jacky Yin

发件人: Jacky Yin 殷传旺 <[hidden email]>
日期: 2019年3月21日星期四下午2:31
收件人: "[hidden email]" <[hidden email]>
主题: Documentation of mesos-appmaster-job.sh

Hello All,

I cannot find any documentation or help about how to use $flin_home/bin/mesos-appmaster-job.sh. Anybody help?

Thanks!

Jacky Yin