state of parallel jobs when one task fails

[Rob]

Hello
I have a simple job with a single map() processing which I want to run with many documents in parallel in Flink.
What will happen if one of the 'instances' of the job fails?
 
This statement in Flink docs confuses me:
"In case of failures, a job switches first to failing where it cancels all running tasks".
So if I have 10 documents processed in parallel in the job's map() (each in a different task slot, I presume) and one of them fails, does it mean that all the rest will be failed/cancelled as well?

Thanks!

[Piotr Nowojski]

Hi,

Yes, by default Flink will restart all of the tasks. I think that since Flink 1.3, you can configure a FailoverStrategy to change this behavior.

Thanks, Piotrek

On Sep 29, 2017, at 5:10 PM, r. r. <[hidden email]> wrote:

Hello
I have a simple job with a single map() processing which I want to run with many documents in parallel in Flink.
What will happen if one of the 'instances' of the job fails?
 
This statement in Flink docs confuses me:
"In case of failures, a job switches first to failing where it cancels all running tasks".
So if I have 10 documents processed in parallel in the job's map() (each in a different task slot, I presume) and one of them fails, does it mean that all the rest will be failed/cancelled as well?

Thanks!

[Rob]

Thanks a lot - wasn't aware of FailoverStrategy

Best regards
Robert








 >-------- Оригинално писмо --------

 >От: Piotr Nowojski [hidden email]

 >Относно: Re: state of parallel jobs when one task fails

 >До: "r. r." <[hidden email]>

 >Изпратено на: 29.09.2017 18:21



 
>
 
>  
 
>  
 
>  
 
>    Hi,
 
>  
 
>  
 
>    
 
>  
 
>  
 
>    Yes, by default Flink will restart all of the tasks. I think that since Flink 1.3, you can configure a 
 
>    FailoverStrategy to change this behavior.
 
>  
 
>  
 
>    
 
>  
 
>  
 
>    Thanks, Piotrek
 
>  
 
>  
 
>  
 
>    
 
>    
 
>      On Sep 29, 2017, at 5:10 PM, r. r. <
 
>      [hidden email]> wrote:
 
>    
 
>    
 
>    
 
>      
 
>       Hello
 
>       I have a simple job with a single map() processing which I want to run with many documents in parallel in Flink.
 
>       What will happen if one of the 'instances' of the job fails?
 
>        
 
>       This statement in Flink docs confuses me:
 
>       "In case of failures, a job switches first to failing where it cancels all running tasks".
 
>       So if I have 10 documents processed in parallel in the job's map() (each in a different task slot, I presume) and one of them fails, does it mean that all the rest will be failed/cancelled as well?
 
>      
 
>       Thanks!
 
>      
 
>      
 
>      
 
>    
 
>    
 
>  
 
>  
 
>