How to test new sink

[Rinat]

Hi !!!

I’ve just implemented a new sink, that extends functionality of existing BucketingSink, currently I’m trying to test functionality, that is related with timing.

My sink implements ProcessingTimeCallback, similarly with the original BucketingSink. I’m trying to inject TestProcessingTimeService to test 

I discovered, that original tests using some kind of test templates for testing functions OneInputStreamOperatorTestHarness, extending which test impl of ProcessingTimeService is injected, 

but this templates are in test scope of flink-streaming-java module, so it could’t be accessed by the external projects.

May be someone could help me and explain the better approach for testing sink functionality ?

Thx !

[Timo Walther]

Hi Rinat,

using one of the Flink test utilities is a good approach to test your custom operators. But of course these classes might change in the future.

First of all, Flink is a open source project so you can just copy the required classes. However, it should be possible to use the Flink test resources like explained here [1]. Flink adds the `test-jar` goal.

@Gary: You are working on the BucketingSink right now, right? Do you have a suggestion?

Regards,
Timo


[1] https://stackoverflow.com/questions/29653914/how-can-i-reference-unit-test-classes-of-a-maven-dependency-in-my-java-project


Am 10/23/17 um 3:16 PM schrieb Rinat:

Hi !!!

I’ve just implemented a new sink, that extends functionality of existing BucketingSink, currently I’m trying to test functionality, that is related with timing.

My sink implements ProcessingTimeCallback, similarly with the original BucketingSink. I’m trying to inject TestProcessingTimeService to test 

I discovered, that original tests using some kind of test templates for testing functions OneInputStreamOperatorTestHarness, extending which test impl of ProcessingTimeService is injected, 

but this templates are in test scope of flink-streaming-java module, so it could’t be accessed by the external projects.

May be someone could help me and explain the better approach for testing sink functionality ?

Thx !

[Rinat]

Timo, thx for your reply.

I’m using gradle instead of maven, but I’ll look through the existing similar plugins for it.

I don’t think, that sharing of external tests between other projects is a good idea, but it’s out of scope of current discussion. 

The main purpose of my request is to understand the existing best practices for testing functions, that uses TimeService and after this, if it’ll be necessary, create an issue with proposals.

Thx.

On 23 Oct 2017, at 17:51, Timo Walther <[hidden email]> wrote:

Hi Rinat,

using one of the Flink test utilities is a good approach to test your custom operators. But of course these classes might change in the future.

First of all, Flink is a open source project so you can just copy the required classes. However, it should be possible to use the Flink test resources like explained here [1]. Flink adds the `test-jar` goal.

@Gary: You are working on the BucketingSink right now, right? Do you have a suggestion?

Regards,
Timo


[1] https://stackoverflow.com/questions/29653914/how-can-i-reference-unit-test-classes-of-a-maven-dependency-in-my-java-project


Am 10/23/17 um 3:16 PM schrieb Rinat:

Hi !!!

I’ve just implemented a new sink, that extends functionality of existing BucketingSink, currently I’m trying to test functionality, that is related with timing.

My sink implements ProcessingTimeCallback, similarly with the original BucketingSink. I’m trying to inject TestProcessingTimeService to test 

I discovered, that original tests using some kind of test templates for testing functions OneInputStreamOperatorTestHarness, extending which test impl of ProcessingTimeService is injected, 

but this templates are in test scope of flink-streaming-java module, so it could’t be accessed by the external projects.

May be someone could help me and explain the better approach for testing sink functionality ?

Thx !

[Timo Walther]

Yes, if you think you need better public test utilities. Feel free to open an issue for it.

Timo


Am 10/23/17 um 5:32 PM schrieb Rinat:

Timo, thx for your reply.

I’m using gradle instead of maven, but I’ll look through the existing similar plugins for it.

I don’t think, that sharing of external tests between other projects is a good idea, but it’s out of scope of current discussion. 

The main purpose of my request is to understand the existing best practices for testing functions, that uses TimeService and after this, if it’ll be necessary, create an issue with proposals.

Thx.

On 23 Oct 2017, at 17:51, Timo Walther <[hidden email]> wrote:

Hi Rinat,

using one of the Flink test utilities is a good approach to test your custom operators. But of course these classes might change in the future.

First of all, Flink is a open source project so you can just copy the required classes. However, it should be possible to use the Flink test resources like explained here [1]. Flink adds the `test-jar` goal.

@Gary: You are working on the BucketingSink right now, right? Do you have a suggestion?

Regards,
Timo


[1] https://stackoverflow.com/questions/29653914/how-can-i-reference-unit-test-classes-of-a-maven-dependency-in-my-java-project


Am 10/23/17 um 3:16 PM schrieb Rinat:

Hi !!!

I’ve just implemented a new sink, that extends functionality of existing BucketingSink, currently I’m trying to test functionality, that is related with timing.

My sink implements ProcessingTimeCallback, similarly with the original BucketingSink. I’m trying to inject TestProcessingTimeService to test 

I discovered, that original tests using some kind of test templates for testing functions OneInputStreamOperatorTestHarness, extending which test impl of ProcessingTimeService is injected, 

but this templates are in test scope of flink-streaming-java module, so it could’t be accessed by the external projects.

May be someone could help me and explain the better approach for testing sink functionality ?

Thx !