Flink error handling

[Steven Nelson]

Hello!

We are internally having a debate on how best to handle exceptions within our operators. Some advocate for wrapping maps/flatMaps inside a processfunction and sending the error to a side output. Other options are returning a custom Either that gets filtered and mapped into different sinks.  

Are there any recommendations or standard solutions to this?

-Steve

[Halfon, Roey]

Hi,
Do you have any progress with that?

-----Original Message-----
From: Steven Nelson <[hidden email]>
Sent: Tuesday, June 18, 2019 7:02 PM
To: [hidden email]
Subject: Flink error handling


Hello!

We are internally having a debate on how best to handle exceptions within our operators. Some advocate for wrapping maps/flatMaps inside a processfunction and sending the error to a side output. Other options are returning a custom Either that gets filtered and mapped into different sinks.  

Are there any recommendations or standard solutions to this?

-Steve

[Steven Nelson]

We ended up using side outputs for now and basically implementing our own map/flatMap that internally uses a ProcessFunction.

Sent from my iPhone

> On Jul 3, 2019, at 6:02 AM, Halfon, Roey <[hidden email]> wrote:
>
> Hi,
> Do you have any progress with that?
>
> -----Original Message-----
> From: Steven Nelson <[hidden email]>
> Sent: Tuesday, June 18, 2019 7:02 PM
> To: [hidden email]
> Subject: Flink error handling
>
>
> Hello!
>
> We are internally having a debate on how best to handle exceptions within our operators. Some advocate for wrapping maps/flatMaps inside a processfunction and sending the error to a side output. Other options are returning a custom Either that gets filtered and mapped into different sinks.  
>
> Are there any recommendations or standard solutions to this?
>
> -Steve