Test harness for validating proper checkpointing of custom SourceFunction

[Ken Krugler]

Hi all,

We’re using the (Keyed)(One|Two)InputStreamOperatorTestHarness classes to test checkpointing of some custom functions.

But in looking through the Flink source, I didn’t see anything comparable for testing a custom SourceFunction (which implements the ListCheckpointed interface).

What’s the recommended approach for this?

We can of course fire up the workflow with checkpointing, and add some additional logic that kills the job after a checkpoint has happened, etc.

But it seems like there should be a better way.

Thanks,

— Ken

--------------------------

Ken Krugler
[phone]
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra

[Aljoscha Krettek]

Hi Ken,

you can use the (slightly misnamed for this purpose) AbstractStreamOperatorTestHarness. It's used in the Flink codebase for unit testing sources.

Best,

Aljoscha

On 12. Sep 2018, at 23:37, Ken Krugler <[hidden email]> wrote:

Hi all,

We’re using the (Keyed)(One|Two)InputStreamOperatorTestHarness classes to test checkpointing of some custom functions.

But in looking through the Flink source, I didn’t see anything comparable for testing a custom SourceFunction (which implements the ListCheckpointed interface).

What’s the recommended approach for this?

We can of course fire up the workflow with checkpointing, and add some additional logic that kills the job after a checkpoint has happened, etc.

But it seems like there should be a better way.

Thanks,

— Ken

--------------------------

Ken Krugler
[phone]
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra