What is the recommended way to run flink with high availability on AWS?

[Pei HE]

Hi there,
Looking at Flink document [1], EMR is the recommended way to run Flink on AWS. However, Flink is currently not supported in an EMR cluster with multiple master nodes [2].

For example, I can create a EMR HA cluster with Zookeeper, Hive, etc. But, if I add Flink to the application list when I create the cluster, I got the following error from aws cli:

"An error occurred (ValidationException) when calling the RunJobFlow operation: HA clusters do not support the specified applications: Flink.  Revise the applications and resubmit.".

My question is what is the recommended way to run flink with high availability on AWS.

Thanks

--

Pei

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html

[2]: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-ha-applications.html

[sri hari kali charan Tummala]

check this out AWS managed Flink cluster in aws kinesis, in AWS kinesis try to create Kinesis data analytics application using flink instead of sql.

On Tue, Aug 20, 2019 at 9:28 PM Pei HE <[hidden email]> wrote:

Hi there,
Looking at Flink document [1], EMR is the recommended way to run Flink on AWS. However, Flink is currently not supported in an EMR cluster with multiple master nodes [2].

For example, I can create a EMR HA cluster with Zookeeper, Hive, etc. But, if I add Flink to the application list when I create the cluster, I got the following error from aws cli:

"An error occurred (ValidationException) when calling the RunJobFlow operation: HA clusters do not support the specified applications: Flink.  Revise the applications and resubmit.".

My question is what is the recommended way to run flink with high availability on AWS.

Thanks

--

Pei

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html

[2]: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-ha-applications.html

--

[Pei HE]

Thanks Kali for the information. However, it doesn't work for me, because I need features in Flink 1.7.x or later and use manged Amazon MSK.

--

Pei

On Tue, Aug 20, 2019 at 7:17 PM sri hari kali charan Tummala <[hidden email]> wrote:

check this out AWS managed Flink cluster in aws kinesis, in AWS kinesis try to create Kinesis data analytics application using flink instead of sql.

On Tue, Aug 20, 2019 at 9:28 PM Pei HE <[hidden email]> wrote:

Hi there,
Looking at Flink document [1], EMR is the recommended way to run Flink on AWS. However, Flink is currently not supported in an EMR cluster with multiple master nodes [2].

For example, I can create a EMR HA cluster with Zookeeper, Hive, etc. But, if I add Flink to the application list when I create the cluster, I got the following error from aws cli:

"An error occurred (ValidationException) when calling the RunJobFlow operation: HA clusters do not support the specified applications: Flink.  Revise the applications and resubmit.".

My question is what is the recommended way to run flink with high availability on AWS.

Thanks

--

Pei

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html

[2]: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-ha-applications.html

--

[sri hari kali charan Tummala]

Ok, no problem. 

On Wed, Aug 21, 2019 at 12:22 AM Pei HE <[hidden email]> wrote:

Thanks Kali for the information. However, it doesn't work for me, because I need features in Flink 1.7.x or later and use manged Amazon MSK.

--

Pei

On Tue, Aug 20, 2019 at 7:17 PM sri hari kali charan Tummala <[hidden email]> wrote:

check this out AWS managed Flink cluster in aws kinesis, in AWS kinesis try to create Kinesis data analytics application using flink instead of sql.

On Tue, Aug 20, 2019 at 9:28 PM Pei HE <[hidden email]> wrote:

Hi there,
Looking at Flink document [1], EMR is the recommended way to run Flink on AWS. However, Flink is currently not supported in an EMR cluster with multiple master nodes [2].

For example, I can create a EMR HA cluster with Zookeeper, Hive, etc. But, if I add Flink to the application list when I create the cluster, I got the following error from aws cli:

"An error occurred (ValidationException) when calling the RunJobFlow operation: HA clusters do not support the specified applications: Flink.  Revise the applications and resubmit.".

My question is what is the recommended way to run flink with high availability on AWS.

Thanks

--

Pei

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html

[2]: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-ha-applications.html

--

--