Unify error handler and late window record output for SQL api

[Yi Tang]

Hi,

I'm looking for a way to handle potential errors in job submitted with SQL API, but unfortunately nothing found.

Handle errors manually in SQL API is hard, I think. Is there a way to handle all errors and send them to a SideOutput to avoid task failure.

Also one can put late records into a SideOutput in streaming API, looks like there's no equivalent in SQL API.

Thanks.

[Yun Gao]

Hi Yi,

      Sorry I'm might not be experts for SQL, as a whole, since SQL should be a high-level API, the users might have less control for the jobs:

      1. Unfortunately we do not have the API to catch all the errors. I think even with DataStream, we also do not provide API to catch  the runtime exception such as the one related to network. Could you also explain a bit on why this functionality is wanted? 

      2. SQL API currently also does not provide API to sideout the late records, since the standard SQL does not provide the corresponding grammar, it would be a bit complex to provide the corresponding functionalities. 

Best,

 Yun

------------------Original Mail ------------------

Sender:Yi Tang <[hidden email]>

Send Date:Thu Oct 29 15:08:30 2020

Recipients:Flink ML <[hidden email]>

Subject:Unify error handler and late window record output for SQL api

Hi,

I'm looking for a way to handle potential errors in job submitted with SQL API, but unfortunately nothing found.

Handle errors manually in SQL API is hard, I think. Is there a way to handle all errors and send them to a SideOutput to avoid task failure.

Also one can put late records into a SideOutput in streaming API, looks like there's no equivalent in SQL API.

Thanks.

[Yi Tang]

Hi Yun
Thanks for your quick reply.

To be clear, It's not essential to implement these features into the SQL
statement. And precisely because of the limitations of SQL, we want these
features happen.

1.
Yeah, I think the stream api also has not similar api. We want it because
sometimes we want to record the error and record itself to analyse instead
of leave it crash the task. Especially in Job by SQL , we can hardly handle
the errors occur in query .
2.
Almost same reason, and also not essential to present the operation in the
statement.

Thanks again.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/