Destroy StreamExecutionEnv

[jay vyas]

Hi folks.

How do we end a stream execution environment? 

I have a unit test which runs a streaming job, and want the unit test to die after the first round of output is processed...

DataStream<Tuple2<Map, Integer>> counts =
    dataStream.map(
        new MapFunction<String, Tuple2<Map, Integer>>() {
          @Override
          public Tuple2<Map, Integer> map(String s) throws Exception {
            Map transaction = MAPPER.readValue(s, Map.class);
            return new Tuple2<>(transaction, 1);
          }
        });
counts.print();

--

[Matthias J. Sax-2]

Hi,

you just need to terminate your source (ie, return from run() method if
you implement your own source function). This will finish the complete
program. For already available sources, just make sure you read finite
input.

Hope this helps.

-Matthias

On 10/05/2015 12:15 AM, jay vyas wrote:

> Hi folks.
>
> How do we end a stream execution environment?
>
> I have a unit test which runs a streaming job, and want the unit test to
> die after the first round of output is processed...
>
>
> DataStream<Tuple2<Map, Integer>> counts =
>     dataStream.map(
>         new MapFunction<String, Tuple2<Map, Integer>>() {
>           @Override
>           public Tuple2<Map, Integer> map(String s) throws Exception {
>             Map transaction = MAPPER.readValue(s, Map.class);
>             return new Tuple2<>(transaction, 1);
>           }
>         });
> counts.print();
>
>
>
> --
> jay vyas

[Stephan Ewen]

Matthias' solution should work in most cases.

In cases where you do not control the source (or the source can never be finite, like the Kafka source), we often use a trick in the tests, which is throwing a special type of exception (a SuccessException).

You can catch this exception on env.execute() (it is the nested cause) and decide that this qualifies the test as successful...

Greetings,
Stephan

On Mon, Oct 5, 2015 at 11:20 AM, Matthias J. Sax <[hidden email]> wrote:

Hi,

you just need to terminate your source (ie, return from run() method if
you implement your own source function). This will finish the complete
program. For already available sources, just make sure you read finite
input.

Hope this helps.

-Matthias

On 10/05/2015 12:15 AM, jay vyas wrote:
> Hi folks.
>
> How do we end a stream execution environment?
>
> I have a unit test which runs a streaming job, and want the unit test to
> die after the first round of output is processed...
>
>
> DataStream<Tuple2<Map, Integer>> counts =
>     dataStream.map(
>         new MapFunction<String, Tuple2<Map, Integer>>() {
>           @Override
>           public Tuple2<Map, Integer> map(String s) throws Exception {
>             Map transaction = MAPPER.readValue(s, Map.class);
>             return new Tuple2<>(transaction, 1);
>           }
>         });
> counts.print();
>
>
>
> --
> jay vyas