ReduceFunction mechanism

[nragon]

Hi,

Regarding ReduceFunction.
Is reduce() called when there is only one record for a given key?

Thanks

[Fabian Hueske-2]

No, the reduce() method of a ReduceFunction requires two elements.

The first received element is just put into state. Once the second element arrives, both are given to the ReduceFunction and the result is put into state and replaces the first element.

Best, Fabian

2017-06-12 18:25 GMT+02:00 nragon <[hidden email]>:

Hi,

Regarding ReduceFunction.
Is reduce() called when there is only one record for a given key?

Thanks



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ReduceFunction-mechanism-tp13651.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[nragon]

So, if my reduce function applies some transformation I must migrate that transformation to a map before the reduce to ensure it transforms, even if there is only one element?
I can chain them together and it will be "almost" as they were in the same function(Ensure same thread processing)?

[Fabian Hueske-2]

You can use a MapFunction (however, it will touch each element and not only the first).
An alternative could be the AggregateFunction if you are using ReduceFunction on a WindowedStream. The interface is a bit more complex though.

Best, Fabian

2017-06-13 10:55 GMT+02:00 nragon <[hidden email]>:

So, if my reduce function applies some transformation I must migrate that
transformation to a map before the reduce to ensure it transforms, even if
there is only one element?
I can chain them together and it will be "almost" as they were in the same
function(Ensure same thread processing)?



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ReduceFunction-mechanism-tp13651p13679.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[nragon]

My goal is to touch each element before the aggregation but i could do it in the reduce function a not having to add another function, thus creating more overhead. The reduce method receives the reduced and a new element which i would change and apply my aggregation.
I'm doing keyby->reduce.
Using a map before all this is a solution.
I've  never tried AggregateFunction, any examples?

Thanks,
Nuno

[nragon]

just an fyi: currently i'm doing map -> keyby -> reduce which in fact could only be keyby -> reduce since reduce can have the map logic.