(DEPRECATED) Apache Flink User Mailing List archive.

Conflict in the document - About native Kubernetes per job mode

classic Classic list List threaded Threaded
♦
♦
 7 messages Options
Fuyao Li-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Conflict in the document - About native Kubernetes per job mode

Fuyao Li-2

Hello Community, Yang,

 

I noticed a conflict in the document for per-job mode support for Kubernetes.

In the doc here [1], it mentions

in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.

It implies per job mode is supported in Kubernetes.

 

However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.

 

This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.

 

I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?

 

 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode

[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s

 

Best,

Fuyao
Yang Wang
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Conflict in the document - About native Kubernetes per job mode

Yang Wang
Hi Fuyao,

Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode
after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode.

In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11
and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in
Kubernetes environment. 


Best,
Yang

Fuyao Li <[hidden email]> 于2021年4月13日周二 上午8:10写道：

Hello Community, Yang,

 

I noticed a conflict in the document for per-job mode support for Kubernetes.

In the doc here [1], it mentions

in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.

It implies per job mode is supported in Kubernetes.

 

However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.

 

This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.

 

I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?

 

 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode

[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s

 

Best,

Fuyao
Fuyao Li-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Fuyao Li-2

Hello Yang,

 

It is very kind of you to give such a detailed explanation! Thanks for clarification.

 

For the small document fix I mentioned, what do you think?

 

Best,

Fuyao

 

From: Yang Wang <[hidden email]>
Date: Monday, April 12, 2021 at 23:03
To: Fuyao Li <[hidden email]>
Cc: user <[hidden email]>, Yan Wang <[hidden email]>
Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Hi Fuyao,

 

Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode

after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode.

 

In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11

and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in

Kubernetes environment. 

 

 

Best,

Yang

 

Fuyao Li <[hidden email]> 于2021年4月13日周二 上午8:10写道：

Hello Community, Yang,

 

I noticed a conflict in the document for per-job mode support for Kubernetes.

In the doc here [1], it mentions

in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.

It implies per job mode is supported in Kubernetes.

 

However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.

 

This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.

 

I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?

 

 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode

[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s

 

Best,

Fuyao
Yang Wang
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Yang Wang
I think it makes sense to have such a simple fix.

Could you please create a ticket and attach a PR?

Best,
Yang

Fuyao Li <[hidden email]> 于2021年4月13日周二 下午2:24写道：

Hello Yang,

 

It is very kind of you to give such a detailed explanation! Thanks for clarification.

 

For the small document fix I mentioned, what do you think?

 

Best,

Fuyao

 

From: Yang Wang <[hidden email]>
Date: Monday, April 12, 2021 at 23:03
To: Fuyao Li <[hidden email]>
Cc: user <[hidden email]>, Yan Wang <[hidden email]>
Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Hi Fuyao,

 

Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode

after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode.

 

In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11

and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in

Kubernetes environment. 

 

 

Best,

Yang

 

Fuyao Li <[hidden email]> 于2021年4月13日周二 上午8:10写道：

Hello Community, Yang,

 

I noticed a conflict in the document for per-job mode support for Kubernetes.

In the doc here [1], it mentions

in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.

It implies per job mode is supported in Kubernetes.

 

However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.

 

This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.

 

I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?

 

 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode

[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s

 

Best,

Fuyao
Fuyao Li-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Fuyao Li-2

Hello Yang,

 

I tried to create a ticket https://issues.apache.org/jira/browse/FLINK-22264

I just registered as a user and I can’t find a place to assign the task to myself… Any idea on this jira issue?

 

Thanks.

 

Best,

Fuyao

 

From: Yang Wang <[hidden email]>
Date: Tuesday, April 13, 2021 at 03:01
To: Fuyao Li <[hidden email]>
Cc: user <[hidden email]>, Yan Wang <[hidden email]>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

I think it makes sense to have such a simple fix.

 

Could you please create a ticket and attach a PR?

 

Best,

Yang

 

Fuyao Li <[hidden email]> 于2021年4月13日周二 下午2:24写道：

Hello Yang,

 

It is very kind of you to give such a detailed explanation! Thanks for clarification.

 

For the small document fix I mentioned, what do you think?

 

Best,

Fuyao

 

From: Yang Wang <[hidden email]>
Date: Monday, April 12, 2021 at 23:03
To: Fuyao Li <[hidden email]>
Cc: user <[hidden email]>, Yan Wang <[hidden email]>
Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Hi Fuyao,

 

Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode

after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode.

 

In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11

and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in

Kubernetes environment. 

 

 

Best,

Yang

 

Fuyao Li <[hidden email]> 于2021年4月13日周二 上午8:10写道：

Hello Community, Yang,

 

I noticed a conflict in the document for per-job mode support for Kubernetes.

In the doc here [1], it mentions

in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.

It implies per job mode is supported in Kubernetes.

 

However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.

 

This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.

 

I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?

 

 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode

[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s

 

Best,

Fuyao
Fuyao Li-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Fuyao Li-2

Hello Yang,

 

I also created a PR for this issue. Please take a look.

Refer to https://github.com/apache/flink/pull/15602

 

Thanks,

Fuyao

 

From: Fuyao Li <[hidden email]>
Date: Tuesday, April 13, 2021 at 18:23
To: Yang Wang <[hidden email]>
Cc: user <[hidden email]>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Hello Yang,

 

I tried to create a ticket https://issues.apache.org/jira/browse/FLINK-22264

I just registered as a user and I can’t find a place to assign the task to myself… Any idea on this jira issue?

 

Thanks.

 

Best,

Fuyao

 

From: Yang Wang <[hidden email]>
Date: Tuesday, April 13, 2021 at 03:01
To: Fuyao Li <[hidden email]>
Cc: user <[hidden email]>, Yan Wang <[hidden email]>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

I think it makes sense to have such a simple fix.

 

Could you please create a ticket and attach a PR?

 

Best,

Yang

 

Fuyao Li <[hidden email]> 于2021年4月13日周二 下午2:24写道：

Hello Yang,

 

It is very kind of you to give such a detailed explanation! Thanks for clarification.

 

For the small document fix I mentioned, what do you think?

 

Best,

Fuyao

 

From: Yang Wang <[hidden email]>
Date: Monday, April 12, 2021 at 23:03
To: Fuyao Li <[hidden email]>
Cc: user <[hidden email]>, Yan Wang <[hidden email]>
Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Hi Fuyao,

 

Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode

after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode.

 

In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11

and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in

Kubernetes environment. 

 

 

Best,

Yang

 

Fuyao Li <[hidden email]> 于2021年4月13日周二 上午8:10写道：

Hello Community, Yang,

 

I noticed a conflict in the document for per-job mode support for Kubernetes.

In the doc here [1], it mentions

in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.

It implies per job mode is supported in Kubernetes.

 

However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.

 

This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.

 

I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?

 

 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode

[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s

 

Best,

Fuyao
Fuyao Li-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Fuyao Li-2

Hello Yang,

 

Please take a look at the PR when you are free.

https://github.com/apache/flink/pull/15602

 

Should be a simple change. Thanks!

 

Best,

Fuyao

 

From: Fuyao Li <[hidden email]>
Date: Tuesday, April 13, 2021 at 19:10
To: Yang Wang <[hidden email]>
Cc: user <[hidden email]>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Hello Yang,

 

I also created a PR for this issue. Please take a look.

Refer to https://github.com/apache/flink/pull/15602

 

Thanks,

Fuyao

 

From: Fuyao Li <[hidden email]>
Date: Tuesday, April 13, 2021 at 18:23
To: Yang Wang <[hidden email]>
Cc: user <[hidden email]>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Hello Yang,

 

I tried to create a ticket https://issues.apache.org/jira/browse/FLINK-22264

I just registered as a user and I can’t find a place to assign the task to myself… Any idea on this jira issue?

 

Thanks.

 

Best,

Fuyao

 

From: Yang Wang <[hidden email]>
Date: Tuesday, April 13, 2021 at 03:01
To: Fuyao Li <[hidden email]>
Cc: user <[hidden email]>, Yan Wang <[hidden email]>
Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

I think it makes sense to have such a simple fix.

 

Could you please create a ticket and attach a PR?

 

Best,

Yang

 

Fuyao Li <[hidden email]> 于2021年4月13日周二 下午2:24写道：

Hello Yang,

 

It is very kind of you to give such a detailed explanation! Thanks for clarification.

 

For the small document fix I mentioned, what do you think?

 

Best,

Fuyao

 

From: Yang Wang <[hidden email]>
Date: Monday, April 12, 2021 at 23:03
To: Fuyao Li <[hidden email]>
Cc: user <[hidden email]>, Yan Wang <[hidden email]>
Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Hi Fuyao,

 

Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode

after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode.

 

In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11

and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in

Kubernetes environment. 

 

 

Best,

Yang

 

Fuyao Li <[hidden email]> 于2021年4月13日周二 上午8:10写道：

Hello Community, Yang,

 

I noticed a conflict in the document for per-job mode support for Kubernetes.

In the doc here [1], it mentions

in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only.

It implies per job mode is supported in Kubernetes.

 

However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes.

 

This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix.

 

I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….?

 

 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode

[4] https://www.youtube.com/watch?v=pdFPr_VOWTU&t=833s

 

Best,

Fuyao
Free forum by Nabble Edit this page