flink on kubernetes

[祁明良]

Hi all,

We are trying to build our flink cluster on k8s, but there seems to be not enough materials about details.

Firstly, we learned how to build a standalone cluster on k8s and run all the jobs inside it.

The question is like are we building a cluster for all the jobs or one job per cluster?

To us, a cluster for all the jobs leads to bad isolation and some job may require different configurations. But we don't find any material talking about how to build like one job per cluster on k8s, something like running one flink job on yarn. Do we have to wrap the things by ourself or there's something we missed?

Best,

Mingliang

本邮件及其附件含有小红书公司的保密信息，仅限于发送给以上收件人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
This communication may contain privileged or other confidential information of Red. If you have received it in error, please advise the sender by reply e-mail and immediately delete the message and any attachments without copying or disclosing the contents. Thank you.

[vino yang]

Hi mingliang,

Yes, you are right, the information that Flink on Kubernetes' current documentation can provide is not very detailed. 

However, considering that Kubernetes is so popular, the Flink community is currently refining it, this work is mainly done by Till, and you can follow this issue [1] to get the latest progress, or you can ask Till if you summarize your list of questions. Regarding the mode of Session mode or single job, I personally recommend that you use the mode of Single Job because it provides better isolation guarantee.

[1]: https://issues.apache.org/jira/browse/FLINK-10001

Thanks, vino.

祁明良 <[hidden email]> 于2018年8月13日周一 上午11:55写道：

Hi all,

We are trying to build our flink cluster on k8s, but there seems to be not enough materials about details.

Firstly, we learned how to build a standalone cluster on k8s and run all the jobs inside it.

The question is like are we building a cluster for all the jobs or one job per cluster?

To us, a cluster for all the jobs leads to bad isolation and some job may require different configurations. But we don't find any material talking about how to build like one job per cluster on k8s, something like running one flink job on yarn. Do we have to wrap the things by ourself or there's something we missed?

Best,

Mingliang

本邮件及其附件含有小红书公司的保密信息，仅限于发送给以上收件人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
This communication may contain privileged or other confidential information of Red. If you have received it in error, please advise the sender by reply e-mail and immediately delete the message and any attachments without copying or disclosing the contents. Thank you.

[Till Rohrmann]

Hi Mingliang,

I'm currently writing the updated documentation for Flink's job cluster container entrypoint. It should be ready later today. In the meantime, you can checkout the `flink-container` module and its subdirectories. They already contain some information in the README.md on how to create a job cluster container image and run it via docker swarm and on K8s.

Cheers,

Till

On Mon, Aug 13, 2018 at 8:23 AM vino yang <[hidden email]> wrote:

Hi mingliang,

Yes, you are right, the information that Flink on Kubernetes' current documentation can provide is not very detailed. 

However, considering that Kubernetes is so popular, the Flink community is currently refining it, this work is mainly done by Till, and you can follow this issue [1] to get the latest progress, or you can ask Till if you summarize your list of questions. Regarding the mode of Session mode or single job, I personally recommend that you use the mode of Single Job because it provides better isolation guarantee.

[1]: https://issues.apache.org/jira/browse/FLINK-10001

Thanks, vino.

祁明良 <[hidden email]> 于2018年8月13日周一 上午11:55写道：

Hi all,

We are trying to build our flink cluster on k8s, but there seems to be not enough materials about details.

Firstly, we learned how to build a standalone cluster on k8s and run all the jobs inside it.

The question is like are we building a cluster for all the jobs or one job per cluster?

To us, a cluster for all the jobs leads to bad isolation and some job may require different configurations. But we don't find any material talking about how to build like one job per cluster on k8s, something like running one flink job on yarn. Do we have to wrap the things by ourself or there's something we missed?

Best,

Mingliang

本邮件及其附件含有小红书公司的保密信息，仅限于发送给以上收件人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
This communication may contain privileged or other confidential information of Red. If you have received it in error, please advise the sender by reply e-mail and immediately delete the message and any attachments without copying or disclosing the contents. Thank you.