Property based testing

[Indraneel R]

Hi All,

Is there any property based testing framework for flink like 'SparkTestingBase'  for spark?

Would also be useful to know what are some of the standard testing practices for data testing for flink pipelines.

regards

-Indraneel

[Aaron Levin]

Hey,

I've used ScalaCheck to test flink applications. Basic idea is:

* use ScalaCheck to generate some kind of collection

* use `fromCollection` in `StreamExecutionEnvironment` to create a `DataStream` 

* use `DataStreamUtils.collect` as a sink

* plug my flink logic between the collection source and the collection sink

* make a ScalaCheck property assertion based on the input collection and output collection.

Possible to wrap all that in a single method in Scala. LMK if you have any more questions or any of this was not clear!

(note: not sure how to do this in Java).

Best,

Aaron Levin

On Wed, Sep 18, 2019 at 8:36 AM Indraneel R <[hidden email]> wrote:

Hi All,

Is there any property based testing framework for flink like 'SparkTestingBase'  for spark?

Would also be useful to know what are some of the standard testing practices for data testing for flink pipelines.

regards

-Indraneel

[Indraneel R]

Oh great! Thanks, Aaron that was quite clear. 

I will give it a try!

On Wed, Sep 18, 2019 at 8:29 PM Aaron Levin <[hidden email]> wrote:

Hey,

I've used ScalaCheck to test flink applications. Basic idea is:

* use ScalaCheck to generate some kind of collection

* use `fromCollection` in `StreamExecutionEnvironment` to create a `DataStream` 

* use `DataStreamUtils.collect` as a sink

* plug my flink logic between the collection source and the collection sink

* make a ScalaCheck property assertion based on the input collection and output collection.

Possible to wrap all that in a single method in Scala. LMK if you have any more questions or any of this was not clear!

(note: not sure how to do this in Java).

Best,

Aaron Levin

On Wed, Sep 18, 2019 at 8:36 AM Indraneel R <[hidden email]> wrote:

Hi All,

Is there any property based testing framework for flink like 'SparkTestingBase'  for spark?

Would also be useful to know what are some of the standard testing practices for data testing for flink pipelines.

regards

-Indraneel