Zookeeper or Kubernetes for HA?

[vishalovercome]

Flink docs provide details on setting up HA but doesn't provide any recommendations as such. For jobs running in kubernetes and having a zookeeper deployment, which high availability option would be more
desirable? Both options have some open bugs and stability would be the most important criteria.

[Matthias]

Hi Vishal,

Do I understand you correctly that you're wondering whether you should stick to ZooKeeper HA on Kubernetes vs Kubernetes HA? You could argue that ZooKeeper might be better since it's already supported for longer and, therefore, better tested. The Kubernetes HA implementation left the experimental stage in 1.12 [1]. There are no major issues I am aware of. Hence, it comes down to the question whether you see the necessity to have an additional component like ZooKeeper in your cluster.

Best,
Matthias

[1] https://flink.apache.org/news/2020/12/10/release-1.12.0.html#kubernetes-high-availability-ha-service

On Mon, May 3, 2021 at 7:13 PM vishalovercome <[hidden email]> wrote:

Flink docs provide details on setting up HA but doesn't provide any
recommendations as such. For jobs running in kubernetes and having a
zookeeper deployment, which high availability option would be more
desirable?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Jack Kolokasis]

Hello,

Does the new release of Flink 1.13.0 includes the cache operation
feature
(https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink).

Thank you,
Iacovos

[Matthias]

Hi Iacovos,

unfortunately, it doesn't as the related FLINK-19343 [1] is not resolved, yet. The release notes for Flink 1.13 can be found in [2].

Best,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-19343

[2] https://flink.apache.org/news/2021/05/03/release-1.13.0.html

On Mon, May 3, 2021 at 8:12 PM Jack Kolokasis <[hidden email]> wrote:

Hello,

Does the new release of Flink 1.13.0 includes the cache operation
feature
(https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink).

Thank you,
Iacovos

[Jack Kolokasis]

Hi Matthias,

Thank you for your reply. Are you going to include it in future versions?

Best,
Iacovos

On 4/5/21 9:10 π.μ., Matthias Pohl wrote:

Hi Iacovos,

unfortunately, it doesn't as the related FLINK-19343 [1] is not resolved, yet. The release notes for Flink 1.13 can be found in [2].

Best,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-19343

[2] https://flink.apache.org/news/2021/05/03/release-1.13.0.html

On Mon, May 3, 2021 at 8:12 PM Jack Kolokasis <[hidden email]> wrote:

Hello,

Does the new release of Flink 1.13.0 includes the cache operation
feature
(https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink).

Thank you,
Iacovos

[Matthias]

It is planned for some future release as the corresponding vote [1] succeeded. I guess the effort is stalled for some time, unfortunately. I'm gonna add Timo to this conversation. He might have more insights on the plans to proceed with it.

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/RESULT-VOTE-FLIP-36-Support-Interactive-Programming-in-Flink-Table-API-td45024.html

On Tue, May 4, 2021 at 8:12 AM Jack Kolokasis <[hidden email]> wrote:

Hi Matthias,

Thank you for your reply. Are you going to include it in future versions?

Best,
Iacovos

On 4/5/21 9:10 π.μ., Matthias Pohl wrote:

Hi Iacovos,

unfortunately, it doesn't as the related FLINK-19343 [1] is not resolved, yet. The release notes for Flink 1.13 can be found in [2].

Best,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-19343

[2] https://flink.apache.org/news/2021/05/03/release-1.13.0.html

On Mon, May 3, 2021 at 8:12 PM Jack Kolokasis <[hidden email]> wrote:

Hello,

Does the new release of Flink 1.13.0 includes the cache operation
feature
(https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink).

Thank you,
Iacovos