Set Readiness, liveness probes on task/job manager pods via Ververica Platform

[narasimha]

Hi, I'm using the ververica platform to host flink jobs. 

Need help in setting up readiness, liveness probes to the taskmanager, jobmanager pods. 

I tried it locally by adding the probe details in deployment.yml file respectively, but it didn't work. 

Can someone help me with setting up the probes. Another question is it possible in the  first place?

--

[Yang Wang]

Do you mean setting the liveness check like the following could not take effect?

        livenessProbe:
          tcpSocket:
            port: 6123
          initialDelaySeconds: 30
          periodSeconds: 60

AFAIK, setting the liveness and the readiness probe is not very necessary for the Flink job. Since

in most cases, the JobManager and TaskManager will exit before the rpc port is not accessible.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午2:08写道：

Hi, I'm using the ververica platform to host flink jobs. 

Need help in setting up readiness, liveness probes to the taskmanager, jobmanager pods. 

I tried it locally by adding the probe details in deployment.yml file respectively, but it didn't work. 

Can someone help me with setting up the probes. Another question is it possible in the  first place?

--

A.Narasimha Swamy

[narasimha]

I have been asked at the org to set it up as per org level standards, so trying to set them. 

As these are health checks with k8s, so that k8s can report if there are any intermittent issues.

Does the JobManager and TaskManager handle failures diligently?

 

On Fri, Feb 5, 2021 at 7:53 AM Yang Wang <[hidden email]> wrote:

Do you mean setting the liveness check like the following could not take effect?

        livenessProbe:
          tcpSocket:
            port: 6123
          initialDelaySeconds: 30
          periodSeconds: 60

AFAIK, setting the liveness and the readiness probe is not very necessary for the Flink job. Since

in most cases, the JobManager and TaskManager will exit before the rpc port is not accessible.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午2:08写道：

Hi, I'm using the ververica platform to host flink jobs. 

Need help in setting up readiness, liveness probes to the taskmanager, jobmanager pods. 

I tried it locally by adding the probe details in deployment.yml file respectively, but it didn't work. 

Can someone help me with setting up the probes. Another question is it possible in the  first place?

--

A.Narasimha Swamy

--

[Yang Wang]

If the JobManager and TaskManager have some fatal errors which they could not correctly handle,

then both of them will directly exit with non-zero code. In such a case, the pod will be restarted.

Once possible scenario I could imagine that the liveness and readiness could help is the long GC.

During the GC period, the rpc port could not be accessed successfully. Also the network issues

could also benefit from the liveness check.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午10:26写道：

I have been asked at the org to set it up as per org level standards, so trying to set them. 

As these are health checks with k8s, so that k8s can report if there are any intermittent issues.

Does the JobManager and TaskManager handle failures diligently?

 

On Fri, Feb 5, 2021 at 7:53 AM Yang Wang <[hidden email]> wrote:

Do you mean setting the liveness check like the following could not take effect?

        livenessProbe:
          tcpSocket:
            port: 6123
          initialDelaySeconds: 30
          periodSeconds: 60

AFAIK, setting the liveness and the readiness probe is not very necessary for the Flink job. Since

in most cases, the JobManager and TaskManager will exit before the rpc port is not accessible.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午2:08写道：

Hi, I'm using the ververica platform to host flink jobs. 

Need help in setting up readiness, liveness probes to the taskmanager, jobmanager pods. 

I tried it locally by adding the probe details in deployment.yml file respectively, but it didn't work. 

Can someone help me with setting up the probes. Another question is it possible in the  first place?

--

A.Narasimha Swamy

--

A.Narasimha Swamy

[narasimha]

Thanks Yang, that was really helpful.

But is there a way to add probes? I could find an example for setup via docker-compose, nothing I could find with VVP.

It will be helpful to have it for the community for other cases as well. Can you please help in setting it up. 

On Fri, Feb 5, 2021 at 8:32 AM Yang Wang <[hidden email]> wrote:

If the JobManager and TaskManager have some fatal errors which they could not correctly handle,

then both of them will directly exit with non-zero code. In such a case, the pod will be restarted.

Once possible scenario I could imagine that the liveness and readiness could help is the long GC.

During the GC period, the rpc port could not be accessed successfully. Also the network issues

could also benefit from the liveness check.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午10:26写道：

I have been asked at the org to set it up as per org level standards, so trying to set them. 

As these are health checks with k8s, so that k8s can report if there are any intermittent issues.

Does the JobManager and TaskManager handle failures diligently?

 

On Fri, Feb 5, 2021 at 7:53 AM Yang Wang <[hidden email]> wrote:

Do you mean setting the liveness check like the following could not take effect?

        livenessProbe:
          tcpSocket:
            port: 6123
          initialDelaySeconds: 30
          periodSeconds: 60

AFAIK, setting the liveness and the readiness probe is not very necessary for the Flink job. Since

in most cases, the JobManager and TaskManager will exit before the rpc port is not accessible.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午2:08写道：

Hi, I'm using the ververica platform to host flink jobs. 

Need help in setting up readiness, liveness probes to the taskmanager, jobmanager pods. 

I tried it locally by adding the probe details in deployment.yml file respectively, but it didn't work. 

Can someone help me with setting up the probes. Another question is it possible in the  first place?

--

A.Narasimha Swamy

--

A.Narasimha Swamy

--

A.Narasimha Swamy

[Yang Wang]

I am not very familiar with ververica platform. But after checking the documentation[1],

I am afraid that setting liveness check could not be supported in VVP.

[1]. https://docs.ververica.com/user_guide/application_operations/deployments/configure_kubernetes.html

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午11:29写道：

Thanks Yang, that was really helpful.

But is there a way to add probes? I could find an example for setup via docker-compose, nothing I could find with VVP.

It will be helpful to have it for the community for other cases as well. Can you please help in setting it up. 

On Fri, Feb 5, 2021 at 8:32 AM Yang Wang <[hidden email]> wrote:

If the JobManager and TaskManager have some fatal errors which they could not correctly handle,

then both of them will directly exit with non-zero code. In such a case, the pod will be restarted.

Once possible scenario I could imagine that the liveness and readiness could help is the long GC.

During the GC period, the rpc port could not be accessed successfully. Also the network issues

could also benefit from the liveness check.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午10:26写道：

I have been asked at the org to set it up as per org level standards, so trying to set them. 

As these are health checks with k8s, so that k8s can report if there are any intermittent issues.

Does the JobManager and TaskManager handle failures diligently?

 

On Fri, Feb 5, 2021 at 7:53 AM Yang Wang <[hidden email]> wrote:

Do you mean setting the liveness check like the following could not take effect?

        livenessProbe:
          tcpSocket:
            port: 6123
          initialDelaySeconds: 30
          periodSeconds: 60

AFAIK, setting the liveness and the readiness probe is not very necessary for the Flink job. Since

in most cases, the JobManager and TaskManager will exit before the rpc port is not accessible.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午2:08写道：

Hi, I'm using the ververica platform to host flink jobs. 

Need help in setting up readiness, liveness probes to the taskmanager, jobmanager pods. 

I tried it locally by adding the probe details in deployment.yml file respectively, but it didn't work. 

Can someone help me with setting up the probes. Another question is it possible in the  first place?

--

A.Narasimha Swamy

--

A.Narasimha Swamy

--

A.Narasimha Swamy

[narasimha]

Thanks Yang for confirming. 

I did try putting in the config, also modifying the deployment.yml in the helm chart. 

 

Adding TIll if this can be taken up.

On Fri, Feb 5, 2021 at 10:37 AM Yang Wang <[hidden email]> wrote:

I am not very familiar with ververica platform. But after checking the documentation[1],

I am afraid that setting liveness check could not be supported in VVP.

[1]. https://docs.ververica.com/user_guide/application_operations/deployments/configure_kubernetes.html

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午11:29写道：

Thanks Yang, that was really helpful.

But is there a way to add probes? I could find an example for setup via docker-compose, nothing I could find with VVP.

It will be helpful to have it for the community for other cases as well. Can you please help in setting it up. 

On Fri, Feb 5, 2021 at 8:32 AM Yang Wang <[hidden email]> wrote:

If the JobManager and TaskManager have some fatal errors which they could not correctly handle,

then both of them will directly exit with non-zero code. In such a case, the pod will be restarted.

Once possible scenario I could imagine that the liveness and readiness could help is the long GC.

During the GC period, the rpc port could not be accessed successfully. Also the network issues

could also benefit from the liveness check.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午10:26写道：

I have been asked at the org to set it up as per org level standards, so trying to set them. 

As these are health checks with k8s, so that k8s can report if there are any intermittent issues.

Does the JobManager and TaskManager handle failures diligently?

 

On Fri, Feb 5, 2021 at 7:53 AM Yang Wang <[hidden email]> wrote:

Do you mean setting the liveness check like the following could not take effect?

        livenessProbe:
          tcpSocket:
            port: 6123
          initialDelaySeconds: 30
          periodSeconds: 60

AFAIK, setting the liveness and the readiness probe is not very necessary for the Flink job. Since

in most cases, the JobManager and TaskManager will exit before the rpc port is not accessible.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午2:08写道：

Hi, I'm using the ververica platform to host flink jobs. 

Need help in setting up readiness, liveness probes to the taskmanager, jobmanager pods. 

I tried it locally by adding the probe details in deployment.yml file respectively, but it didn't work. 

Can someone help me with setting up the probes. Another question is it possible in the  first place?

--

A.Narasimha Swamy

--

A.Narasimha Swamy

--

A.Narasimha Swamy

--

[Fabian Paul]

We are currently working on supporting arbitrary pod template specs for the 

Flink pods. It allows you to specify a V1PodTemplateSpec for taskmanager 

and jobmanager.

The feature will be included in the next upcoming release 2.4 of the 

ververica platform. We plan to release it in the next few months.

Best,

Fabian

On 5. Feb 2021, at 06:23, narasimha <[hidden email]> wrote:

Thanks Yang for confirming. 

I did try putting in the config, also modifying the deployment.yml in the helm chart. 

 

Adding TIll if this can be taken up.

On Fri, Feb 5, 2021 at 10:37 AM Yang Wang <[hidden email]> wrote:

I am not very familiar with ververica platform. But after checking the documentation[1],

I am afraid that setting liveness check could not be supported in VVP.

[1]. https://docs.ververica.com/user_guide/application_operations/deployments/configure_kubernetes.html

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午11:29写道：

Thanks Yang, that was really helpful.

But is there a way to add probes? I could find an example for setup via docker-compose, nothing I could find with VVP.

It will be helpful to have it for the community for other cases as well. Can you please help in setting it up. 

On Fri, Feb 5, 2021 at 8:32 AM Yang Wang <[hidden email]> wrote:

If the JobManager and TaskManager have some fatal errors which they could not correctly handle,

then both of them will directly exit with non-zero code. In such a case, the pod will be restarted.

Once possible scenario I could imagine that the liveness and readiness could help is the long GC.

During the GC period, the rpc port could not be accessed successfully. Also the network issues

could also benefit from the liveness check.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午10:26写道：

I have been asked at the org to set it up as per org level standards, so trying to set them. 

As these are health checks with k8s, so that k8s can report if there are any intermittent issues.

Does the JobManager and TaskManager handle failures diligently?

 

On Fri, Feb 5, 2021 at 7:53 AM Yang Wang <[hidden email]> wrote:

Do you mean setting the liveness check like the following could not take effect?

        livenessProbe:
          tcpSocket:
            port: 6123
          initialDelaySeconds: 30
          periodSeconds: 60

AFAIK, setting the liveness and the readiness probe is not very necessary for the Flink job. Since

in most cases, the JobManager and TaskManager will exit before the rpc port is not accessible.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午2:08写道：

Hi, I'm using the ververica platform to host flink jobs. 

Need help in setting up readiness, liveness probes to the taskmanager, jobmanager pods. 

I tried it locally by adding the probe details in deployment.yml file respectively, but it didn't work. 

Can someone help me with setting up the probes. Another question is it possible in the  first place?

--

A.Narasimha Swamy

--

A.Narasimha Swamy

--

A.Narasimha Swamy

--

[narasimha]

Great, thanks for the update. 

On Fri, Feb 5, 2021 at 2:06 PM Fabian Paul <[hidden email]> wrote:

We are currently working on supporting arbitrary pod template specs for the 

Flink pods. It allows you to specify a V1PodTemplateSpec for taskmanager 

and jobmanager.

The feature will be included in the next upcoming release 2.4 of the 

ververica platform. We plan to release it in the next few months.

Best,

Fabian

On 5. Feb 2021, at 06:23, narasimha <[hidden email]> wrote:

Thanks Yang for confirming. 

I did try putting in the config, also modifying the deployment.yml in the helm chart. 

 

Adding TIll if this can be taken up.

On Fri, Feb 5, 2021 at 10:37 AM Yang Wang <[hidden email]> wrote:

I am not very familiar with ververica platform. But after checking the documentation[1],

I am afraid that setting liveness check could not be supported in VVP.

[1]. https://docs.ververica.com/user_guide/application_operations/deployments/configure_kubernetes.html

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午11:29写道：

Thanks Yang, that was really helpful.

But is there a way to add probes? I could find an example for setup via docker-compose, nothing I could find with VVP.

It will be helpful to have it for the community for other cases as well. Can you please help in setting it up. 

On Fri, Feb 5, 2021 at 8:32 AM Yang Wang <[hidden email]> wrote:

If the JobManager and TaskManager have some fatal errors which they could not correctly handle,

then both of them will directly exit with non-zero code. In such a case, the pod will be restarted.

Once possible scenario I could imagine that the liveness and readiness could help is the long GC.

During the GC period, the rpc port could not be accessed successfully. Also the network issues

could also benefit from the liveness check.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午10:26写道：

I have been asked at the org to set it up as per org level standards, so trying to set them. 

As these are health checks with k8s, so that k8s can report if there are any intermittent issues.

Does the JobManager and TaskManager handle failures diligently?

 

On Fri, Feb 5, 2021 at 7:53 AM Yang Wang <[hidden email]> wrote:

Do you mean setting the liveness check like the following could not take effect?

        livenessProbe:
          tcpSocket:
            port: 6123
          initialDelaySeconds: 30
          periodSeconds: 60

AFAIK, setting the liveness and the readiness probe is not very necessary for the Flink job. Since

in most cases, the JobManager and TaskManager will exit before the rpc port is not accessible.

Best,

Yang

narasimha <[hidden email]> 于2021年2月5日周五 上午2:08写道：

Hi, I'm using the ververica platform to host flink jobs. 

Need help in setting up readiness, liveness probes to the taskmanager, jobmanager pods. 

I tried it locally by adding the probe details in deployment.yml file respectively, but it didn't work. 

Can someone help me with setting up the probes. Another question is it possible in the  first place?

--

A.Narasimha Swamy

--

A.Narasimha Swamy

--

A.Narasimha Swamy

--

A.Narasimha Swamy

--