Re: Counting the number of elements in a dataset

Posted by Stefan Bunk on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Counting-the-number-of-elements-in-a-dataset-tp437p440.html

If it's about the verbosity, you can just use iter.size instead of your self-written count, right?

val numVertices =

    (srcVertices union targetVertices).distinct.reduceGroup { iter => iter.size }

Performance-wise, this is the same, though.

Cheers

Stefan

On Sat, Nov 22, 2014 at 8:17 PM, Márton Balassi <[hidden email]> wrote:

Hey,

There was a thread recently on the dev list that might be interesting to you [1].

I do not know the exact state of the code though.

[1] http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Hi-Aggregation-support-td2311.html#a2547

Cheers,

Marton

On Sat, Nov 22, 2014 at 8:09 PM, Sebastian Schelter <[hidden email]> wrote:

Hi,

Is there a simple way to count the number of elements of a dataset? At the moment, I have to use the following code, which is pretty verbose and unefficient.

    val numVertices =
      (srcVertices union targetVertices).distinct.reduceGroup { iter =>
        var count = 1L
        while (iter.hasNext) {
          count += 1
          iter.next
        }
        count
      }

Best,
Sebastian