Re: cogroup

Posted by Fabian Hueske-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/cogroup-tp1839p1841.html

If you just want to do the pairwise comparison try join().
Join is an inner join and will give you all pairs of elements with matching keys.

For CoGroup, there is no other way than collecting one side in memory.

Best, Fabian

2015-06-29 17:42 GMT+02:00 Matthias J. Sax <[hidden email]>:

Why do you not use a join? CoGroup seems not to be the right operator.

-Matthias

On 06/29/2015 05:40 PM, Michele Bertoni wrote:
> Hi I have a question on cogroup
>
> when I cogroup two dataset is there a way to compare each element on the left with each element on the right (inside a group) without collecting one side?
>
> right now I am doing
>
> left.cogroup(right).where(0,1,2).equalTo(0,1,2){
>       (leftIterator, rightIterator, out) => {
>               val lSet = leftIterator.toSet           // <———— toSet
>               for(r <- rightIterator)
>                       for(l <- lSet)
>                               //do something
>       }
> }
>
> I would like to avoid the toSet
>
>
> thanks for help
>