test (harness and minicluster)

[Martin Frank Hansen]

Hi, 

I am trying to make a test-suite for our flink cluster using harness and minicluster. As we are using processWindowFunctions in our pipeline we need some good ways of validating these functions. To my surprise processWindowFunctions are neither supported by test-harness or minicluster setups, so does anyone know if it will be supported in the future? (Or if it is supported how should I use it?) 

Furthermore does anyone have some good ideas for a test-setup for processWindowFunctions? 

best regards

[Till Rohrmann]

Hi Martin,

In general, Flink's MiniCluster should be able to run every complete Flink JobGraph. However, from what I read you are looking for a test harness for a processWindowFunction so that you can test this function in a more unit test style, right? What you can do is to use the OneInputStreamOperatorTestHarness and initialize it with a WindowOperator where you pass the ProcessWindowFunction to. That way you should be able to test the ProcessWindowFunction. Please also take a look at org.apache.flink.streaming.runtime.operators.windowing.WindowOperatorTest [1] for some examples how it could be done.

[1] https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java

Cheers,

Till

On Wed, Dec 2, 2020 at 12:04 PM Martin Frank Hansen <[hidden email]> wrote:

Hi, 

I am trying to make a test-suite for our flink cluster using harness and minicluster. As we are using processWindowFunctions in our pipeline we need some good ways of validating these functions. To my surprise processWindowFunctions are neither supported by test-harness or minicluster setups, so does anyone know if it will be supported in the future? (Or if it is supported how should I use it?) 

Furthermore does anyone have some good ideas for a test-setup for processWindowFunctions? 

best regards

Martin Frank Hansen

Data Engineer

[Martin Frank Hansen]

Hi Till, 
Thanks for your answer! 
I will try the approach you suggested, hope I can make it work.

Best regards
martin

Den ons. 2. dec. 2020 kl. 17.03 skrev Till Rohrmann <[hidden email]>:

Hi Martin,

In general, Flink's MiniCluster should be able to run every complete Flink JobGraph. However, from what I read you are looking for a test harness for a processWindowFunction so that you can test this function in a more unit test style, right? What you can do is to use the OneInputStreamOperatorTestHarness and initialize it with a WindowOperator where you pass the ProcessWindowFunction to. That way you should be able to test the ProcessWindowFunction. Please also take a look at org.apache.flink.streaming.runtime.operators.windowing.WindowOperatorTest [1] for some examples how it could be done.

[1] https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java

Cheers,

Till

On Wed, Dec 2, 2020 at 12:04 PM Martin Frank Hansen <[hidden email]> wrote:

Hi, 

I am trying to make a test-suite for our flink cluster using harness and minicluster. As we are using processWindowFunctions in our pipeline we need some good ways of validating these functions. To my surprise processWindowFunctions are neither supported by test-harness or minicluster setups, so does anyone know if it will be supported in the future? (Or if it is supported how should I use it?) 

Furthermore does anyone have some good ideas for a test-setup for processWindowFunctions? 

best regards

Martin Frank Hansen

Data Engineer

--