Monitoring job w/LocalStreamEnvironment

[Ken Krugler]

Hi all,

With an iteration-based workflow, it’s helpful to be able to monitor the job counters and explicitly terminate when the test has completed.

I didn’t see support for async job creation, though.

So I extended LocalStreamEnvironment to add an executeAsync(), which returns the LocalFlinkMiniCluster.submitJobDetached() result.

But it appears like I need to have a ClusterClient in order to actually monitor this job.

And ClusterClient is bound in with a lot of CLI code, so I’m hesitant to try to extract what I need.

Is there an easier/recommended approach to the above?

Thanks!

— Ken

--------------------------------------------
http://about.me/kkrugler
[phone]

[Piotr Nowojski]

Hi,

Have you read the following doc?

https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html

There are some hints regarding testing your application. Especially take a look at the example with using static field to communicate with the running job.

Piotrek

On 12 Oct 2017, at 16:33, Ken Krugler <[hidden email]> wrote:

Hi all,

With an iteration-based workflow, it’s helpful to be able to monitor the job counters and explicitly terminate when the test has completed.

I didn’t see support for async job creation, though.

So I extended LocalStreamEnvironment to add an executeAsync(), which returns the LocalFlinkMiniCluster.submitJobDetached() result.

But it appears like I need to have a ClusterClient in order to actually monitor this job.

And ClusterClient is bound in with a lot of CLI code, so I’m hesitant to try to extract what I need.

Is there an easier/recommended approach to the above?

Thanks!

— Ken

--------------------------------------------
http://about.me/kkrugler
[phone]

[Piotr Nowojski]

Hi,

Regarding metrics please check the "Writing an Integration test for flink-metrics” recent mailing list question. You can either use JMXReporter or write some custom reporter for this purpose.

Piotrek

On 13 Oct 2017, at 20:57, Ken Krugler <[hidden email]> wrote:

Hi Piotr,

Thanks for responding, see below.

On Oct 12, 2017, at 7:51 AM, Piotr Nowojski <[hidden email]> wrote:

Hi,

Have you read the following doc?

https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html

There are some hints regarding testing your application. Especially take a look at the example with using static field to communicate with the running job.

Yes, I’d read those.

I already have a bunch of Flink metrics, I was hoping to leverage those to know when my test can safely terminate my iteration;

I guess I could create a metrics wrapper that also logs to a static class during tests.

Regards,

— Ken

On 12 Oct 2017, at 16:33, Ken Krugler <[hidden email]> wrote:

Hi all,

With an iteration-based workflow, it’s helpful to be able to monitor the job counters and explicitly terminate when the test has completed.

I didn’t see support for async job creation, though.

So I extended LocalStreamEnvironment to add an executeAsync(), which returns the LocalFlinkMiniCluster.submitJobDetached() result.

But it appears like I need to have a ClusterClient in order to actually monitor this job.

And ClusterClient is bound in with a lot of CLI code, so I’m hesitant to try to extract what I need.

Is there an easier/recommended approach to the above?

Thanks!

— Ken

--------------------------------------------
http://about.me/kkrugler
[phone]

--------------------------------------------

http://about.me/kkrugler

[phone]