test windows
Locked
 
|
Hi all,
How can I test in Java any streaming job that has a time window? best, Nicos |
|
|
Hi Nicos, Under the flink-example module, there are many examples, including batch and streaming. You could build the project from the source, this way you could found many jars under the target directory. You can submit these jars to the Flink cluster. Also, you could run these examples directly from your IDE by standalone mode. best, Zhengwen On Tue, Aug 28, 2018 at 7:35 PM Nicos Maris <[hidden email]> wrote:
_________________________________________
Zhu Zhengwen State Key Laboratory For Novel Software Technology Dept. of Computer Sci. & Tech., Nanjing University 9, Hankou Road, Nanjing, China. |
|
|
Hi Zhengwen, I have tested my job manually (both by submitting a job and through execute()) and I am trying to write a test. The following project states that it has the feature "Test stream windowing with timestamped input" but I do not want to rely on a project other than flink. The following folder doesn't seam to demonstrate such capability for testing. On Tue, Aug 28, 2018 at 2:55 PM ZhengWen ZHU <[hidden email]> wrote:
|
|
|
Can the OneInputStreamOperatorTestHarness be used to test pipelines with time windows? Can someone explain this test harness preferably with a tiny example? I see here that "it is currently not a part of public API and can be subject to change", but I assume it is the only thing I can get from flink as regards automated testing of pipelines with time windows. On Tue, Aug 28, 2018 at 3:12 PM Nicos Maris <[hidden email]> wrote:
|
|
|
Hi Nicos, Do you want an IT test to test the job, or just a UT to test the window operator? If you want to test the job, there are examples here: Also, there are UT tests using test harness to test a window operator: Best, Hequn On Thu, Aug 30, 2018 at 9:04 PM Nicos Maris <[hidden email]> wrote:
|
|
|
In reply to this post by Nicos Maris
The flink training exercises have a simpler example of using a TwoInputStreamOperatorTestHarness from outside of the Flink code base that you can refer to. The two input test harness is more or less the same as the one input test harness.
You'll need these dependencies: <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-test-utils-junit</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>${flink.version}</version> <scope>test</scope> <type>test-jar</type> </dependency> <dependency> <groupId>org.mockito</groupId> <artifactId>mockito-all</artifactId> <version>1.10.19</version> <type>jar</type> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-runtime_2.11</artifactId> <version>${flink.version}</version> <scope>test</scope> <type>test-jar</type> </dependency> On Thu, Aug 30, 2018 at 3:04 PM Nicos Maris <[hidden email]> wrote: > > Can the OneInputStreamOperatorTestHarness be used to test pipelines with time windows? > > Can someone explain this test harness preferably with a tiny example? > > I see here that "it is currently not a part of public API and can be subject to change", but I assume it is the only thing I can get from flink as regards automated testing of pipelines with time windows. > > On Tue, Aug 28, 2018 at 3:12 PM Nicos Maris <[hidden email]> wrote: >> >> Hi Zhengwen, >> >> >> I have tested my job manually (both by submitting a job and through execute()) and I am trying to write a test. >> >> The following project states that it has the feature "Test stream windowing with timestamped input" but I do not want to rely on a project other than flink. >> >> https://github.com/ottogroup/flink-spector >> >> The following folder doesn't seam to demonstrate such capability for testing. >> >> https://github.com/apache/flink/tree/master/flink-examples/flink-examples-streaming/src/test/java/org/apache/flink/streaming/test >> >> >> >> On Tue, Aug 28, 2018 at 2:55 PM ZhengWen ZHU <[hidden email]> wrote: >>> >>> Hi Nicos, >>> Under the flink-example module, there are many examples, including batch and streaming. You could build the project from the source, this way you could found many jars under the target directory. You can submit these jars to the Flink cluster. Also, you could run these examples directly from your IDE by standalone mode. >>> >>> best, >>> Zhengwen >>> >>> On Tue, Aug 28, 2018 at 7:35 PM Nicos Maris <[hidden email]> wrote: >>>> >>>> Hi all, >>>> >>>> >>>> How can I test in Java any streaming job that has a time window? >>>> >>>> >>>> best, >>>> Nicos >>> >>> >>> >>> -- >>> _________________________________________ >>> >>> Zhu Zhengwen >>> State Key Laboratory For Novel Software Technology >>> Dept. of Computer Sci. & Tech., Nanjing University >>> 9, Hankou Road, Nanjing, China. -- David Anderson | Training Coordinator | data Artisans -- Join Flink Forward - The Apache Flink Conference Stream Processing | Event Driven | Real Time |
|
|
Hi all, It took me some time to reply. I am focusing on testing my job, not on unit tests and the SourceFunction as in WindowFunctionITCase.scala has helped me a lot :) Is there any test with keyed windows? I am writing in java with POJOs instead of tuples. On Fri, Aug 31, 2018 at 3:43 PM David Anderson <[hidden email]> wrote:
|
|
|
Hi Nicos, I think this is what you want. WindowFoldITCase. :-) Flink is a open source project, you can checkout the source code from git and search the tests in it. For example, you can search the class by name that contains 'window'. There are a lot of other test cases which I think will help you. Best, Hequn On Sun, Sep 9, 2018 at 10:29 AM Nicos Maris <[hidden email]> wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |