flink testing

[Georg Heiler]

Hi,

is there something like spark-testing-base for flink as well?

Cheers,

Georg

[Ted Yu]

Please give more context by describing what spark-test-base does :-)

> On Apr 22, 2017, at 10:57 PM, Georg Heiler <[hidden email]> wrote:
>
> Hi,
>
> is there something like spark-testing-base for flink as well?
>
> Cheers,
> Georg

[Georg Heiler]

Spark testing base https://github.com/holdenk/spark-testing-base offers some Base classes to use when writing tests with Spark which make it simpler to write unit tests for functions in spark i.e. you do not manually need to instantiate a spark context / flink execution environment for each test case.

You've written an awesome program in Spark and now its time to write some tests. Only you find yourself writing the code to setup and tear down local mode Spark in between each suite and you say to your self: This is not my beautiful code.

Ted Yu <[hidden email]> schrieb am So., 23. Apr. 2017 um 10:46 Uhr:

Please give more context by describing what spark-test-base does :-)

> On Apr 22, 2017, at 10:57 PM, Georg Heiler <[hidden email]> wrote:
>
> Hi,
>
> is there something like spark-testing-base for flink as well?
>
> Cheers,
> Georg

[Maximilian Bode]

Hi Georg,

Have a look at (Streaming)MultipleProgramsTestBase (https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/StreamingMultipleProgramsTestBase.java | https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/MultipleProgramsTestBase.java).

Best regards,

Max

Am 23.04.2017 um 17:19 schrieb Georg Heiler <[hidden email]>:

Spark testing base https://github.com/holdenk/spark-testing-base offers some Base classes to use when writing tests with Spark which make it simpler to write unit tests for functions in spark i.e. you do not manually need to instantiate a spark context / flink execution environment for each test case.

You've written an awesome program in Spark and now its time to write some tests. Only you find yourself writing the code to setup and tear down local mode Spark in between each suite and you say to your self: This is not my beautiful code.

Ted Yu <[hidden email]> schrieb am So., 23. Apr. 2017 um 10:46 Uhr:

Please give more context by describing what spark-test-base does :-)

> On Apr 22, 2017, at 10:57 PM, Georg Heiler <[hidden email]> wrote:
>
> Hi,
>
> is there something like spark-testing-base for flink as well?
>
> Cheers,
> Georg

[snntr]

+ user@flink (somehow I didn't reply to the list)

Hi Georg,

there are two classes "MultipleProgramsTestBase" and
"StreamingMultipeProgramsTestBase", which your integration tests can
extend from.

This will spin up a local Flink Mini-Cluster before the test execution
and tear it down afterwards.

I have also recently build a small Flink-JUnit rule available on Github
to make this a little bit easier and  more flexible [1]. The snapshot
version should be available in the sonatype open source repositories in
the next days. Let me know, if it seems useful to you.

This is for integration tests of our jobs or parts of the jobs though.
For unit tests we usually mock the RuntimeContext.

Cheers,

Konstantin

[1] https://github.com/knaufk/flink-junit

On 23.04.2017 17:19, Georg Heiler wrote:

> Spark testing base https://github.com/holdenk/spark-testing-base offers
> some Base classes to use when writing tests with Spark which make it
> simpler to write unit tests for functions in spark i.e. you do not
> manually need to instantiate a spark context / flink execution
> environment for each test case.
>
> You've written an awesome program in Spark and now its time to write
> some tests. Only you find yourself writing the code to setup and tear
> down local mode Spark in between each suite and you say to your self:
> This is not my beautiful code.
>
>
>     <https://github.com/holdenk/spark-testing-base#how>
>
>
> Ted Yu <[hidden email] <mailto:[hidden email]>> schrieb am
> So., 23. Apr. 2017 um 10:46 Uhr:
>
>     Please give more context by describing what spark-test-base does :-)
>
>     > On Apr 22, 2017, at 10:57 PM, Georg Heiler
>     <[hidden email] <mailto:[hidden email]>> wrote:
>     >
>     > Hi,
>     >
>     > is there something like spark-testing-base for flink as well?
>     >
>     > Cheers,
>     > Georg
>

--
Konstantin Knauf * [hidden email] * [phone]
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082

[Georg Heiler]

Thanks

Konstantin Knauf <[hidden email]> schrieb am So. 23. Apr. 2017 um 19:39:

+ user@flink (somehow I didn't reply to the list)

Hi Georg,

there are two classes "MultipleProgramsTestBase" and
"StreamingMultipeProgramsTestBase", which your integration tests can
extend from.

This will spin up a local Flink Mini-Cluster before the test execution
and tear it down afterwards.

I have also recently build a small Flink-JUnit rule available on Github
to make this a little bit easier and  more flexible [1]. The snapshot
version should be available in the sonatype open source repositories in
the next days. Let me know, if it seems useful to you.

This is for integration tests of our jobs or parts of the jobs though.
For unit tests we usually mock the RuntimeContext.

Cheers,

Konstantin

[1] https://github.com/knaufk/flink-junit

On 23.04.2017 17:19, Georg Heiler wrote:
> Spark testing base https://github.com/holdenk/spark-testing-base offers
> some Base classes to use when writing tests with Spark which make it
> simpler to write unit tests for functions in spark i.e. you do not
> manually need to instantiate a spark context / flink execution
> environment for each test case.
>
> You've written an awesome program in Spark and now its time to write
> some tests. Only you find yourself writing the code to setup and tear
> down local mode Spark in between each suite and you say to your self:
> This is not my beautiful code.
>
>
>     <https://github.com/holdenk/spark-testing-base#how>
>
>
> Ted Yu <[hidden email] <mailto:[hidden email]>> schrieb am
> So., 23. Apr. 2017 um 10:46 Uhr:
>
>     Please give more context by describing what spark-test-base does :-)
>
>     > On Apr 22, 2017, at 10:57 PM, Georg Heiler
>     <[hidden email] <mailto:[hidden email]>> wrote:
>     >
>     > Hi,
>     >
>     > is there something like spark-testing-base for flink as well?
>     >
>     > Cheers,
>     > Georg
>

--
Konstantin Knauf * [hidden email] * [phone]
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082

[Fabian Hueske-2]

Another option might be Flink Spector [1].

Cheers, Fabian

[1] https://github.com/ottogroup/flink-spector

2017-04-23 20:01 GMT+02:00 Georg Heiler <[hidden email]>:

Thanks

Konstantin Knauf <[hidden email]> schrieb am So. 23. Apr. 2017 um 19:39:

+ user@flink (somehow I didn't reply to the list)

Hi Georg,

there are two classes "MultipleProgramsTestBase" and
"StreamingMultipeProgramsTestBase", which your integration tests can
extend from.

This will spin up a local Flink Mini-Cluster before the test execution
and tear it down afterwards.

I have also recently build a small Flink-JUnit rule available on Github
to make this a little bit easier and  more flexible [1]. The snapshot
version should be available in the sonatype open source repositories in
the next days. Let me know, if it seems useful to you.

This is for integration tests of our jobs or parts of the jobs though.
For unit tests we usually mock the RuntimeContext.

Cheers,

Konstantin

[1] https://github.com/knaufk/flink-junit

On 23.04.2017 17:19, Georg Heiler wrote:
> Spark testing base https://github.com/holdenk/spark-testing-base offers
> some Base classes to use when writing tests with Spark which make it
> simpler to write unit tests for functions in spark i.e. you do not
> manually need to instantiate a spark context / flink execution
> environment for each test case.
>
> You've written an awesome program in Spark and now its time to write
> some tests. Only you find yourself writing the code to setup and tear
> down local mode Spark in between each suite and you say to your self:
> This is not my beautiful code.
>
>
>     <https://github.com/holdenk/spark-testing-base#how>
>
>
> Ted Yu <[hidden email] <mailto:[hidden email]>> schrieb am
> So., 23. Apr. 2017 um 10:46 Uhr:
>
>     Please give more context by describing what spark-test-base does :-)
>
>     > On Apr 22, 2017, at 10:57 PM, Georg Heiler
>     <[hidden email] <mailto:[hidden email]>> wrote:
>     >
>     > Hi,
>     >
>     > is there something like spark-testing-base for flink as well?
>     >
>     > Cheers,
>     > Georg
>

--
Konstantin Knauf * [hidden email] * <a href="tel:+49%20174%203413182" value="[phone]" target="_blank">[phone]
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082