FlatMap on a batch of elements

[Flavio Pompermaier]

Hy all,

in my FlatMap function I'd like to call a service every 100 elements. 

That would be possible if Collector was passed to the close function but it is not.

How can I simulate that?

[Stefan Bunk]

Hi,

maybe mapPartition ("Transforms a parallel partition in a single function call. The function get the partition as an `Iterator` and can produce an arbitrary number of result values.") is the right thing for you?

Cheers

Stefan

On Fri, Nov 21, 2014 at 10:08 AM, Flavio Pompermaier <[hidden email]> wrote:

Hy all,

in my FlatMap function I'd like to call a service every 100 elements. 

That would be possible if Collector was passed to the close function but it is not.

How can I simulate that?

[Flavio Pompermaier]

Yes, that's perfectly fine!

Thanks!

On Fri, Nov 21, 2014 at 10:14 AM, Stefan Bunk <[hidden email]> wrote:

Hi,

maybe mapPartition ("Transforms a parallel partition in a single function call. The function get the partition as an `Iterator` and can produce an arbitrary number of result values.") is the right thing for you?

Cheers

Stefan

On Fri, Nov 21, 2014 at 10:08 AM, Flavio Pompermaier <[hidden email]> wrote:

Hy all,

in my FlatMap function I'd like to call a service every 100 elements. 

That would be possible if Collector was passed to the close function but it is not.

How can I simulate that?