Flink

[Navneeth Krishnan]

Hi All,

I'm very new to EKS and trying to deploy a flink job in cluster mode. Are there any good documentations on what are the steps to deploy on EKS?

From my understanding, with flink 1.10 running it on EKS will automatically scale up and down with kubernetes integration based on the load. Is this correct? Do I have to do enable some configs to support this feature?

How to use the lyft k8s operator when deploying on EKS?

Thanks a lot, appreciate all the help.

[Timo Walther]

Hi Navneeth,

it might be also worth to look into Ververica Plaform for this. The
community edition was published recently is free of charge. It provides
first class K8s support [1].

There is also a tutorial how to deploy it on EKS [2] (not the most
recent one through).

Regards,
Timo

[1]
https://www.ververica.com/blog/announcing-ververica-platform-community-edition?utm_campaign=Ververica%20Platform%20-%20Community%20Edition&utm_content=123140986&utm_medium=social&utm_source=twitter&hss_channel=tw-2581958070
[2]
https://www.ververica.com/blog/how-to-get-started-with-data-artisans-platform-on-aws-eks



On 15.04.20 03:57, Navneeth Krishnan wrote:

> Hi All,
>
> I'm very new to EKS and trying to deploy a flink job in cluster mode.
> Are there any good documentations on what are the steps to deploy on EKS?
>
>  From my understanding, with flink 1.10 running it on EKS will
> automatically scale up and down with kubernetes integration based on the
> load. Is this correct? Do I have to do enable some configs to support
> this feature?
>
> How to use the lyft k8s operator when deploying on EKS?
>
> Thanks a lot, appreciate all the help.
>
>
>
>

[Navneeth Krishnan]

Thanks a lot Timo. I will take a look at it. But does flink automatically scale up and down at this point with native integration? 

Thanks

On Tue, Apr 14, 2020 at 11:27 PM Timo Walther <[hidden email]> wrote:

Hi Navneeth,

it might be also worth to look into Ververica Plaform for this. The
community edition was published recently is free of charge. It provides
first class K8s support [1].

There is also a tutorial how to deploy it on EKS [2] (not the most
recent one through).

Regards,
Timo

[1]
https://www.ververica.com/blog/announcing-ververica-platform-community-edition?utm_campaign=Ververica%20Platform%20-%20Community%20Edition&utm_content=123140986&utm_medium=social&utm_source=twitter&hss_channel=tw-2581958070
[2]
https://www.ververica.com/blog/how-to-get-started-with-data-artisans-platform-on-aws-eks



On 15.04.20 03:57, Navneeth Krishnan wrote:
> Hi All,
>
> I'm very new to EKS and trying to deploy a flink job in cluster mode.
> Are there any good documentations on what are the steps to deploy on EKS?
>
>  From my understanding, with flink 1.10 running it on EKS will
> automatically scale up and down with kubernetes integration based on the
> load. Is this correct? Do I have to do enable some configs to support
> this feature?
>
> How to use the lyft k8s operator when deploying on EKS?
>
> Thanks a lot, appreciate all the help.
>
>
>
>

[Konstantin Knauf-2]

Hi Navneeth,

I think there might be some misunderstanding here. Let me try to clarify.

1) The so-called native Kubernetes support [1], which was added as an experimental feature in Flink 1.10, is not used by Ververica Platform CE nor by the Lyft K8s Operator as far as I am aware of.

2) The native Kubernetes support only supports session clusters [1] so far. Support for Job clusters work in progress for Flink 1.11. In case of a session cluster, Flink will request additional resources from Kubernetes whenever necessary to deploy a Flink Job upon submission. So, there is automatic scaling of the cluster, but no automatic scaling of the jobs.

3) There are multiple efforts within the Flink ecosystem working on auto scaling jobs.

I hope this helps and please reach out if you have further questions.

Best,

Konstantin

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.10/concepts/glossary.html#flink-session-cluster

On Thu, Apr 23, 2020 at 7:11 AM Navneeth Krishnan <[hidden email]> wrote:

Thanks a lot Timo. I will take a look at it. But does flink automatically scale up and down at this point with native integration? 

Thanks

On Tue, Apr 14, 2020 at 11:27 PM Timo Walther <[hidden email]> wrote:

Hi Navneeth,

it might be also worth to look into Ververica Plaform for this. The
community edition was published recently is free of charge. It provides
first class K8s support [1].

There is also a tutorial how to deploy it on EKS [2] (not the most
recent one through).

Regards,
Timo

[1]
https://www.ververica.com/blog/announcing-ververica-platform-community-edition?utm_campaign=Ververica%20Platform%20-%20Community%20Edition&utm_content=123140986&utm_medium=social&utm_source=twitter&hss_channel=tw-2581958070
[2]
https://www.ververica.com/blog/how-to-get-started-with-data-artisans-platform-on-aws-eks



On 15.04.20 03:57, Navneeth Krishnan wrote:
> Hi All,
>
> I'm very new to EKS and trying to deploy a flink job in cluster mode.
> Are there any good documentations on what are the steps to deploy on EKS?
>
>  From my understanding, with flink 1.10 running it on EKS will
> automatically scale up and down with kubernetes integration based on the
> load. Is this correct? Do I have to do enable some configs to support
> this feature?
>
> How to use the lyft k8s operator when deploying on EKS?
>
> Thanks a lot, appreciate all the help.
>
>
>
>

--

Konstantin Knauf | Head of Product

[phone]

Follow us @VervericaData Ververica

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Tony) Cheng