UT/IT Table API/SQL streaming job

[narasimha]

Hi, 

I'm checking on how to do UT/IT of streaming job written using Table API/SQL. 

I found https://stackoverflow.com/questions/54900843/add-a-unit-test-for-flink-sql this to be useful. 

Are there any other recommended libs/ways to do this. 

TIA

--

[Timo Walther]

Hi,

this might be helpful as well:

https://lists.apache.org/thread.html/rfe3b45a10fc58cf19d2f71c6841515eb7175ba731d5055b06f236b3f%40%3Cuser.flink.apache.org%3E

First of all, it is important to know if you are interested in
end-to-end tests (incl. connectors) or excluding connectors. If you just
like to test you operators, you can use a lot of the testing
infrastructure of Flink.

If your are NOT using event-time, you can simply use
`TableEnvironment.fromValues()` and `Table.execute().collect()`. This
test uses it for example [1] (it is one of the newer test generations).

Otherwise you can use or implement your own testing connectors, like in
org.apache.flink.table.planner.runtime.stream.sql.FunctionITCase#testStructuredScalarFunction
[2].

I hope this helps.

Regards,
Timo

[1]
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/expressions/MathFunctionsITCase.java

[2]
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/runtime/stream/sql/FunctionITCase.java#L700

On 19.08.20 12:43, narasimha wrote:

> Hi,
>
> I'm checking on how to do UT/IT of streaming job written using Table
> API/SQL.
>
> I found
> https://stackoverflow.com/questions/54900843/add-a-unit-test-for-flink-sql this
> to be useful.
>
> Are there any other recommended libs/ways to do this.
>
> TIA
> --
> A.Narasimha Swamy

[Till Rohrmann]

Hi Narasimha,

have you taken a look at the testing documentation [1]? This should give you some pointers on how to test Flink jobs.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html

Cheers,

Till

On Wed, Aug 19, 2020 at 12:44 PM narasimha <[hidden email]> wrote:

Hi, 

I'm checking on how to do UT/IT of streaming job written using Table API/SQL. 

I found https://stackoverflow.com/questions/54900843/add-a-unit-test-for-flink-sql this to be useful. 

Are there any other recommended libs/ways to do this. 

TIA

--

A.Narasimha Swamy

[narasimha]

Hi Till, 

Yes, I have gone through the Flink testing documentation. 

In Table API/SQL connectors can be abstracted in the query itself, trying to understand how such pipelines can be tested. 

Looking for resources around it.  

On Wed, Aug 19, 2020 at 5:15 PM Till Rohrmann <[hidden email]> wrote:

Hi Narasimha,

have you taken a look at the testing documentation [1]? This should give you some pointers on how to test Flink jobs.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html

Cheers,

Till

On Wed, Aug 19, 2020 at 12:44 PM narasimha <[hidden email]> wrote:

Hi, 

I'm checking on how to do UT/IT of streaming job written using Table API/SQL. 

I found https://stackoverflow.com/questions/54900843/add-a-unit-test-for-flink-sql this to be useful. 

Are there any other recommended libs/ways to do this. 

TIA

--

A.Narasimha Swamy

--

[narasimha]

Thanks Timo,

I'll check it out.

On Wed, Aug 19, 2020 at 5:14 PM Timo Walther <[hidden email]> wrote:

Hi,

this might be helpful as well:

https://lists.apache.org/thread.html/rfe3b45a10fc58cf19d2f71c6841515eb7175ba731d5055b06f236b3f%40%3Cuser.flink.apache.org%3E

First of all, it is important to know if you are interested in
end-to-end tests (incl. connectors) or excluding connectors. If you just
like to test you operators, you can use a lot of the testing
infrastructure of Flink.

If your are NOT using event-time, you can simply use
`TableEnvironment.fromValues()` and `Table.execute().collect()`. This
test uses it for example [1] (it is one of the newer test generations).

Otherwise you can use or implement your own testing connectors, like in
org.apache.flink.table.planner.runtime.stream.sql.FunctionITCase#testStructuredScalarFunction
[2].

I hope this helps.

Regards,
Timo

[1]
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/expressions/MathFunctionsITCase.java

[2]
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/runtime/stream/sql/FunctionITCase.java#L700

On 19.08.20 12:43, narasimha wrote:
> Hi,
>
> I'm checking on how to do UT/IT of streaming job written using Table
> API/SQL.
>
> I found
> https://stackoverflow.com/questions/54900843/add-a-unit-test-for-flink-sql this
> to be useful.
>
> Are there any other recommended libs/ways to do this.
>
> TIA
> --
> A.Narasimha Swamy

--