Report failed job submission

[Flavio Pompermaier]

Hi to all,

which is the correct wat to report back to the user a failure from a job submission in FLink?

If everything is OK the job run API returns the job id but what if there are error in parameter validation or some other problem? 

Which is the right way to report back to the user the job error detail (apart from throwing an Exception)?

Best,

Flavio

[Flavio Pompermaier]

Any idea about how to address this issue?

On Tue, Oct 16, 2018 at 11:32 AM Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

which is the correct wat to report back to the user a failure from a job submission in FLink?

If everything is OK the job run API returns the job id but what if there are error in parameter validation or some other problem? 

Which is the right way to report back to the user the job error detail (apart from throwing an Exception)?

Best,

Flavio

[Timo Walther]

Hi Flavio,

I'm not entirely sure if I get your question correct but what you are looking for is more information (like categorization) why the submission failed right?

Regards,
Timo


Am 06.11.18 um 14:33 schrieb Flavio Pompermaier:

Any idea about how to address this issue?

On Tue, Oct 16, 2018 at 11:32 AM Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

which is the correct wat to report back to the user a failure from a job submission in FLink?

If everything is OK the job run API returns the job id but what if there are error in parameter validation or some other problem? 

Which is the right way to report back to the user the job error detail (apart from throwing an Exception)?

Best,

Flavio

[Flavio Pompermaier]

Let's say that my job needs to do some check before running (like existence of a file or some other condition): at the moment I can only throw an Exception but on the client side you get only something like: {"errors":["org.apache.flink.client.program.ProgramInvocationException: The main method caused an error."]}

I was wondering if there is any better way to handle this kind of problems..

On Mon, Nov 12, 2018 at 3:53 PM Timo Walther <[hidden email]> wrote:

Hi Flavio,

I'm not entirely sure if I get your question correct but what you are looking for is more information (like categorization) why the submission failed right?

Regards,
Timo


Am 06.11.18 um 14:33 schrieb Flavio Pompermaier:

Any idea about how to address this issue?

On Tue, Oct 16, 2018 at 11:32 AM Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

which is the correct wat to report back to the user a failure from a job submission in FLink?

If everything is OK the job run API returns the job id but what if there are error in parameter validation or some other problem? 

Which is the right way to report back to the user the job error detail (apart from throwing an Exception)?

Best,

Flavio

[Timo Walther]

I assume you are using the REST API? Flink's RestClusterClient is able to deserialize the exception including its cause that might be more helpful in your case as well.

The entire exception should be queryable via execution result [1]. At least we get a better error using the SQL Client [2] as an example.

Regards,
Timo

[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/rest_api.html#jobs-jobid-execution-result
[2] https://github.com/apache/flink/blob/master/flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ProgramDeployer.java


Am 12.11.18 um 16:00 schrieb Flavio Pompermaier:

Let's say that my job needs to do some check before running (like existence of a file or some other condition): at the moment I can only throw an Exception but on the client side you get only something like: {"errors":["org.apache.flink.client.program.ProgramInvocationException: The main method caused an error."]}

I was wondering if there is any better way to handle this kind of problems..

On Mon, Nov 12, 2018 at 3:53 PM Timo Walther <[hidden email]> wrote:

Hi Flavio,

I'm not entirely sure if I get your question correct but what you are looking for is more information (like categorization) why the submission failed right?

Regards,
Timo


Am 06.11.18 um 14:33 schrieb Flavio Pompermaier:

Any idea about how to address this issue?

On Tue, Oct 16, 2018 at 11:32 AM Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

which is the correct wat to report back to the user a failure from a job submission in FLink?

If everything is OK the job run API returns the job id but what if there are error in parameter validation or some other problem? 

Which is the right way to report back to the user the job error detail (apart from throwing an Exception)?

Best,

Flavio