Counting the number of elements in a dataset

[Sebastian Schelter-2]

Hi,

Is there a simple way to count the number of elements of a dataset? At
the moment, I have to use the following code, which is pretty verbose
and unefficient.

     val numVertices =
       (srcVertices union targetVertices).distinct.reduceGroup { iter =>
         var count = 1L
         while (iter.hasNext) {
           count += 1
           iter.next
         }
         count
       }

Best,
Sebastian

[Márton Balassi]

Hey,

There was a thread recently on the dev list that might be interesting to you [1].

I do not know the exact state of the code though.

[1] http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Hi-Aggregation-support-td2311.html#a2547

Cheers,

Marton

On Sat, Nov 22, 2014 at 8:09 PM, Sebastian Schelter <[hidden email]> wrote:

Hi,

Is there a simple way to count the number of elements of a dataset? At the moment, I have to use the following code, which is pretty verbose and unefficient.

    val numVertices =
      (srcVertices union targetVertices).distinct.reduceGroup { iter =>
        var count = 1L
        while (iter.hasNext) {
          count += 1
          iter.next
        }
        count
      }

Best,
Sebastian

[Stefan Bunk]

If it's about the verbosity, you can just use iter.size instead of your self-written count, right?

val numVertices =

    (srcVertices union targetVertices).distinct.reduceGroup { iter => iter.size }

Performance-wise, this is the same, though.

Cheers

Stefan

On Sat, Nov 22, 2014 at 8:17 PM, Márton Balassi <[hidden email]> wrote:

Hey,

There was a thread recently on the dev list that might be interesting to you [1].

I do not know the exact state of the code though.

[1] http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Hi-Aggregation-support-td2311.html#a2547

Cheers,

Marton

On Sat, Nov 22, 2014 at 8:09 PM, Sebastian Schelter <[hidden email]> wrote:

Hi,

Is there a simple way to count the number of elements of a dataset? At the moment, I have to use the following code, which is pretty verbose and unefficient.

    val numVertices =
      (srcVertices union targetVertices).distinct.reduceGroup { iter =>
        var count = 1L
        while (iter.hasNext) {
          count += 1
          iter.next
        }
        count
      }

Best,
Sebastian