Local collection data sink for the streaming API

[Filipe Correia]

Hi,

Collecting results locally (e.g., for unit testing) is possible in the
DataSet API by using "LocalCollectionOutputFormat", as described in
the programming guide:
https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/programming_guide.html#collection-data-sources-and-sinks

Can something similar be done for the DataStream API?

Thanks,

Filipe

[Gábor Gévay]

Hi Filipe,

You can take a look at `DataStreamUtils.collect` in
flink-contrib/flink-streaming-contrib.

Best,
Gábor



2016-01-05 16:14 GMT+01:00 Filipe Correia <[hidden email]>:

> Hi,
>
> Collecting results locally (e.g., for unit testing) is possible in the
> DataSet API by using "LocalCollectionOutputFormat", as described in
> the programming guide:
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/programming_guide.html#collection-data-sources-and-sinks
>
> Can something similar be done for the DataStream API?
>
> Thanks,
>
> Filipe

[Filipe Correia]

Hi Gábor, Thanks!

I'm using Scala though. DataStreamUtils.collect() depends on
org.apache.flink.streaming.api.datastream.DataStream, rather than
org.apache.flink.streaming.api.scala.DataStream. Any suggestion on how
to handle this, other than creating my own scala implementation of
DataStreamUtils.collect()?

Thanks,

Filipe

On Tue, Jan 5, 2016 at 3:33 PM, Gábor Gévay <[hidden email]> wrote:

> Hi Filipe,
>
> You can take a look at `DataStreamUtils.collect` in
> flink-contrib/flink-streaming-contrib.
>
> Best,
> Gábor
>
>
>
> 2016-01-05 16:14 GMT+01:00 Filipe Correia <[hidden email]>:
>> Hi,
>>
>> Collecting results locally (e.g., for unit testing) is possible in the
>> DataSet API by using "LocalCollectionOutputFormat", as described in
>> the programming guide:
>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/programming_guide.html#collection-data-sources-and-sinks
>>
>> Can something similar be done for the DataStream API?
>>
>> Thanks,
>>
>> Filipe

[Gábor Gévay]

Try the getJavaStream method of the scala DataStream.

Best,
Gábor




2016-01-05 19:14 GMT+01:00 Filipe Correia <[hidden email]>:

> Hi Gábor, Thanks!
>
> I'm using Scala though. DataStreamUtils.collect() depends on
> org.apache.flink.streaming.api.datastream.DataStream, rather than
> org.apache.flink.streaming.api.scala.DataStream. Any suggestion on how
> to handle this, other than creating my own scala implementation of
> DataStreamUtils.collect()?
>
> Thanks,
>
> Filipe
>
> On Tue, Jan 5, 2016 at 3:33 PM, Gábor Gévay <[hidden email]> wrote:
>> Hi Filipe,
>>
>> You can take a look at `DataStreamUtils.collect` in
>> flink-contrib/flink-streaming-contrib.
>>
>> Best,
>> Gábor
>>
>>
>>
>> 2016-01-05 16:14 GMT+01:00 Filipe Correia <[hidden email]>:
>>> Hi,
>>>
>>> Collecting results locally (e.g., for unit testing) is possible in the
>>> DataSet API by using "LocalCollectionOutputFormat", as described in
>>> the programming guide:
>>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/programming_guide.html#collection-data-sources-and-sinks
>>>
>>> Can something similar be done for the DataStream API?
>>>
>>> Thanks,
>>>
>>> Filipe

[Filipe Correia]

Perfect, thanks!

Filipe

On Tue, Jan 5, 2016 at 6:23 PM, Gábor Gévay <[hidden email]> wrote:

> Try the getJavaStream method of the scala DataStream.
>
> Best,
> Gábor
>
>
>
>
> 2016-01-05 19:14 GMT+01:00 Filipe Correia <[hidden email]>:
>> Hi Gábor, Thanks!
>>
>> I'm using Scala though. DataStreamUtils.collect() depends on
>> org.apache.flink.streaming.api.datastream.DataStream, rather than
>> org.apache.flink.streaming.api.scala.DataStream. Any suggestion on how
>> to handle this, other than creating my own scala implementation of
>> DataStreamUtils.collect()?
>>
>> Thanks,
>>
>> Filipe
>>
>> On Tue, Jan 5, 2016 at 3:33 PM, Gábor Gévay <[hidden email]> wrote:
>>> Hi Filipe,
>>>
>>> You can take a look at `DataStreamUtils.collect` in
>>> flink-contrib/flink-streaming-contrib.
>>>
>>> Best,
>>> Gábor
>>>
>>>
>>>
>>> 2016-01-05 16:14 GMT+01:00 Filipe Correia <[hidden email]>:
>>>> Hi,
>>>>
>>>> Collecting results locally (e.g., for unit testing) is possible in the
>>>> DataSet API by using "LocalCollectionOutputFormat", as described in
>>>> the programming guide:
>>>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/programming_guide.html#collection-data-sources-and-sinks
>>>>
>>>> Can something similar be done for the DataStream API?
>>>>
>>>> Thanks,
>>>>
>>>> Filipe