RestartStrategy failure count when losing a Task Manager

[Jiahui Jiang]

Hello Flink, I have some questions regarding to the guideline on configuring restart strategy.

I was testing a job with the following setup:

1. There are many tasks, but currently I'm running with only 2 parallelism, but plenty of task slots (4 TM and 4 task slot in each TM).
   
2. It's ran in k8s with HA enabled.
3. The current restart strategy is 'failure-rate' with 30mins failure interval, 1 min delay interval and 3 failure rate.

When a TM got removed by k8s, it looked like that caused multiple failure to happen all at once. In the job manager log, I'm seeing different task failed with the same stacktrace 'Heartbeat of taskManager with id {SOME_ID} timed out' around the same time.

I understand that all the tasks that were running on this taskManager would fail. But still have these following questions:

Questions:

1. What count as one failure for a restartStrategy? It doesn't look like every failed task counts towards one failure according to my other jobs. Is it because we have failover strategy defaults to be region, and each failure only trigger part of the job graph to restart, and the rest of the 'not retriggered' job graph can still cause more failure that will be counted towards failure rate?
   
2. If that's the case, what will be the recommended way to set restart strategy? If we don't want to hard code a number for every single pipeline we are running, is that a good way to infer how to set the failure rate?

Thank you so much!

Jiahui

[Chesnay Schepler]

1) A restart in one region only increments the count by 1, independent of how many tasks from that region fail at the same time.

If tasks from different regions fail at the same time, then the bound is incremented by the number of affected regions.

2)

I would consider what failure rate is acceptable if there were no regions, and then multiple that but the number of slots to offset task executor failures.

Failures in the application (e.g., a source failing for some reason) will generally behave, failure-rate wise, as if regions would not exist. They are sporadic, and the chance of them appearing in different regions at the same time seems rather small.

On 15/07/2020 00:16, Jiahui Jiang wrote:

Hello Flink, I have some questions regarding to the guideline on configuring restart strategy.

I was testing a job with the following setup:

1. There are many tasks, but currently I'm running with only 2 parallelism, but plenty of task slots (4 TM and 4 task slot in each TM).
   
2. It's ran in k8s with HA enabled.
3. The current restart strategy is 'failure-rate' with 30mins failure interval, 1 min delay interval and 3 failure rate.

When a TM got removed by k8s, it looked like that caused multiple failure to happen all at once. In the job manager log, I'm seeing different task failed with the same stacktrace 'Heartbeat of taskManager with id {SOME_ID} timed out' around the same time.

I understand that all the tasks that were running on this taskManager would fail. But still have these following questions:

Questions:

1. What count as one failure for a restartStrategy? It doesn't look like every failed task counts towards one failure according to my other jobs. Is it because we have failover strategy defaults to be region, and each failure only trigger part of the job graph to restart, and the rest of the 'not retriggered' job graph can still cause more failure that will be counted towards failure rate?
   
2. If that's the case, what will be the recommended way to set restart strategy? If we don't want to hard code a number for every single pipeline we are running, is that a good way to infer how to set the failure rate?

Thank you so much!

Jiahui