Python API + Unit Testing

[Kevin Lam]

Hi all,

I noticed there isn't much in the way of testing discussed in the Python API docs for Flink.

Does the community have any best-practices or recommendations on how testing should be done with PyFlink? 

Thanks!

[Dian Fu]

Hi,

Do you mean how to run Python unit test? If so, you could refer to [1] for more details.

Regards,

Dian

[1] https://cwiki.apache.org/confluence/display/FLINK/Setting+up+a+Flink+development+environment

2021年3月18日 下午10:46，Kevin Lam <[hidden email]> 写道：

Hi all,

I noticed there isn't much in the way of testing discussed in the Python API docs for Flink.

Does the community have any best-practices or recommendations on how testing should be done with PyFlink? 

Thanks!

[Kevin Lam]

Hi Dian Fu, 

I meant testing in application development. When I'm developing a Pyflink Pipeline, are there any recommended approaches to testing the Flink application? For instance, how should we test applications end-to-end? Individual operators? 

I'm interested in the Datastream API. 

One approach I could see is using StreamingFileSinks, and validating the output files for a bounded stream. 

On Thu, Mar 18, 2021 at 10:04 PM Dian Fu <[hidden email]> wrote:

Hi,

Do you mean how to run Python unit test? If so, you could refer to [1] for more details.

Regards,

Dian

[1] https://cwiki.apache.org/confluence/display/FLINK/Setting+up+a+Flink+development+environment

2021年3月18日 下午10:46，Kevin Lam <[hidden email]> 写道：

Hi all,

I noticed there isn't much in the way of testing discussed in the Python API docs for Flink.

Does the community have any best-practices or recommendations on how testing should be done with PyFlink? 

Thanks!

... [show rest of quote]

[Dian Fu]

Hi Kevin,

>> One approach I could see is using StreamingFileSinks, and validating the output files for a bounded stream. 

This is a good approach from my point of view.  Actually, the end to end tests in Flink also takes this kind of approach. This doesn't only apply for Python jobs, but also for Java / Scala / SQL jobs.

Regards,
Dian

On Fri, Mar 19, 2021 at 10:37 PM Kevin Lam <[hidden email]> wrote:

Hi Dian Fu, 

I meant testing in application development. When I'm developing a Pyflink Pipeline, are there any recommended approaches to testing the Flink application? For instance, how should we test applications end-to-end? Individual operators? 

I'm interested in the Datastream API. 

One approach I could see is using StreamingFileSinks, and validating the output files for a bounded stream. 

On Thu, Mar 18, 2021 at 10:04 PM Dian Fu <[hidden email]> wrote:

Hi,

Do you mean how to run Python unit test? If so, you could refer to [1] for more details.

Regards,

Dian

[1] https://cwiki.apache.org/confluence/display/FLINK/Setting+up+a+Flink+development+environment

2021年3月18日 下午10:46，Kevin Lam <[hidden email]> 写道：

Hi all,

I noticed there isn't much in the way of testing discussed in the Python API docs for Flink.

Does the community have any best-practices or recommendations on how testing should be done with PyFlink? 

Thanks!

... [show rest of quote]

... [show rest of quote]