Unit/Integration test for stateful source

[Darshan Singh]

Hi,

I am writing a stateful source very similar to KafkaBaseConsumer but not as generic.  I was looking on how we can use unit test cases and integration tests on this. I looked at the kafka-connector-based unit test cases. It seems that there is too much external things at play here like lots of mock classes created and then even Abstract fetcher was created. 

So I sort of dropped the idea where I can check if snapshot is basically calling offset clear as well as store the offsets. 

But doing the integration tests is hard as well especially as I want to see if it picks up offsets properly. I can run it manually and see it but I was hoping if we can get some directions on integration tests for stateful source.

Thanks

[Stefan Richter]

Hi,

maybe you can use AbstractStreamOperatorTestHarness to test your source, including the snapshotting. You can take a look at the tests of some other source, e.g. StatefulSequenceSourceTest#testCheckpointRestore.

Best,
Stefan

> Am 20.09.2018 um 15:29 schrieb Darshan Singh <[hidden email]>:
>
> Hi,
>
> I am writing a stateful source very similar to KafkaBaseConsumer but not as generic.  I was looking on how we can use unit test cases and integration tests on this. I looked at the kafka-connector-based unit test cases. It seems that there is too much external things at play here like lots of mock classes created and then even Abstract fetcher was created.
>
> So I sort of dropped the idea where I can check if snapshot is basically calling offset clear as well as store the offsets.
>
> But doing the integration tests is hard as well especially as I want to see if it picks up offsets properly. I can run it manually and see it but I was hoping if we can get some directions on integration tests for stateful source.
>
> Thanks