Testing Metrics

[Julio Biason]

Hey guys and gals,

Just wondering: Does anyone have an idea how to test if metrics are being generated? I have an integration test and I just added a processor to count elements in late arrivals (the general idea is to capture those, count and get an average, so we can adjust the allowedLateness) but now I'm wondering if there is a way I can integrate this into the test itself.

Possible? Not possible? Ideas?

--

[Tzu-Li (Gordon) Tai]

Hi,

Do you mean tests to verify that some metric is actually registered?

AFAIK, this is not really easy to do as a unit test.

One possible way is to have an integration test that uses a metrics reporter, from which you verify against.

For example, the Kafka consumer integration tests that uses the JMXReporter for a submitted job and queries results from that to verify that some specific metrics are forwarded from the Kafka client.

We also have an end-to-end test for savepointing, that uses the SLF4JReporter to write metrics to logs, which are parsed for a specific metric to monitor progress.

Also pulling in Chesnay (in CC), who might have more ideas here.

Cheers,

Gordon

On 25 April 2018 at 1:59:18 AM, Julio Biason ([hidden email]) wrote:

Hey guys and gals,

Just wondering: Does anyone have an idea how to test if metrics are being generated? I have an integration test and I just added a processor to count elements in late arrivals (the general idea is to capture those, count and get an average, so we can adjust the allowedLateness) but now I'm wondering if there is a way I can integrate this into the test itself.

Possible? Not possible? Ideas?

--

[Chesnay Schepler]

+1 to using reporters.

You will have to explicitly pass a configuration with the reporter settings to the environment via StreamExecutionEnvironment#createLocalEnvironment(int, Configuration).

The reporter can verify registrations/values and pass this information back to the main test thread through a static field (for simplicity).

On 25.04.2018 09:11, Tzu-Li (Gordon) Tai wrote:

Hi,

Do you mean tests to verify that some metric is actually registered?

AFAIK, this is not really easy to do as a unit test.

One possible way is to have an integration test that uses a metrics reporter, from which you verify against.

For example, the Kafka consumer integration tests that uses the JMXReporter for a submitted job and queries results from that to verify that some specific metrics are forwarded from the Kafka client.

We also have an end-to-end test for savepointing, that uses the SLF4JReporter to write metrics to logs, which are parsed for a specific metric to monitor progress.

Also pulling in Chesnay (in CC), who might have more ideas here.

Cheers,

Gordon

On 25 April 2018 at 1:59:18 AM, Julio Biason ([hidden email]) wrote:

Hey guys and gals,

Just wondering: Does anyone have an idea how to test if metrics are being generated? I have an integration test and I just added a processor to count elements in late arrivals (the general idea is to capture those, count and get an average, so we can adjust the allowedLateness) but now I'm wondering if there is a way I can integrate this into the test itself.

Possible? Not possible? Ideas?

--