Fwd: Flink - Exception Handling best practices

[Michel Betancourt]

Hi, new to Apache Flink.  Trying to find some solid input on how best to handle exceptions in streams -- specifically those that should not interrupt the stream.  

For example, if an error occurs during deserialization from bytes/Strings to your data-type, in my use-case I would rather queue the data for visual inspection than discard it and filter it out.

One way of doing this is to diverge the stream so that good items take one path, while bad items take another.  

The closest thing I can find in Flink that can achieve this effect is the split operator. The caveat is that split does not also allow for inlined transformations.  In other words, the best use of split appears first perform your logic that catches the exception.  Then pass the exception into the next stage which uses split to check for an exception and providing names to each piece of the decision, for example "OK" vs "error".

Frameworks like RX (Reactive Extensions, eg RxJava) have built in functionality that allows the user to decide if they want to handle exceptions globally or specifically and resume if needed.  I was hoping to find similar operations in Flink but so far no luck.  

At any rate, it would be great to get some feedback to see if I am heading down the good path here, and whether there are any caveats / gotchas to be aware of?

Thanks!

Mich

[Fabian Hueske-2]

Hi Mich,

at the moment there is not much support handle such data driven exceptions (badly formatted data, late data, ...).

However, there is a proposal to improve this: FLIP-13 [1]. So it is work in progress.

It would be very helpful if you could check if the proposal would address your use cases.

Until then, I guess that either the split operator or directly writing to an external storage system would be the way to go.

Best,

Fabian

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-13+Side+Outputs+in+Flink

2016-11-11 2:33 GMT+01:00 Michel Betancourt <[hidden email]>:

Hi, new to Apache Flink.  Trying to find some solid input on how best to handle exceptions in streams -- specifically those that should not interrupt the stream.  

For example, if an error occurs during deserialization from bytes/Strings to your data-type, in my use-case I would rather queue the data for visual inspection than discard it and filter it out.

One way of doing this is to diverge the stream so that good items take one path, while bad items take another.  

The closest thing I can find in Flink that can achieve this effect is the split operator. The caveat is that split does not also allow for inlined transformations.  In other words, the best use of split appears first perform your logic that catches the exception.  Then pass the exception into the next stage which uses split to check for an exception and providing names to each piece of the decision, for example "OK" vs "error".

Frameworks like RX (Reactive Extensions, eg RxJava) have built in functionality that allows the user to decide if they want to handle exceptions globally or specifically and resume if needed.  I was hoping to find similar operations in Flink but so far no luck.  

At any rate, it would be great to get some feedback to see if I am heading down the good path here, and whether there are any caveats / gotchas to be aware of?

Thanks!

Mich