(DEPRECATED) Apache Flink User Mailing List archive.

Incremental aggregations - Example not working

classic Classic list List threaded Threaded
♦
♦
 8 messages Options
Matt
Reply | Threaded
Open this post in threaded view
♦
♦
|

Incremental aggregations - Example not working

Matt
Hi,

I'm following the documentation [1] of window functions with incremental aggregations, but I'm getting an "input mismatch" error.

The code [2] is almost identical to the one in the documentation, at the bottom you can find the exact error.

What am I missing? Can you provide a working example of a fold function with both a FoldFunction and a WindowFunction?

Regards,
Matt


[2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
Matt
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Incremental aggregations - Example not working

Matt
In case this is important, if I remove the WindowFunction, and only use the FoldFunction it works fine.

I don't see what is wrong...

On Mon, Dec 12, 2016 at 10:53 AM, Matt <[hidden email]> wrote:
Hi,

I'm following the documentation [1] of window functions with incremental aggregations, but I'm getting an "input mismatch" error.

The code [2] is almost identical to the one in the documentation, at the bottom you can find the exact error.

What am I missing? Can you provide a working example of a fold function with both a FoldFunction and a WindowFunction?

Regards,
Matt


[2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
Chesnay Schepler
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Incremental aggregations - Example not working

Chesnay Schepler
Hello Matt,

This looks like a bug in the fold() function to me.

I'm adding Timo to the discussion, he can probably shed some light on this.

Regards,
Chesnay

On 12.12.2016 15:13, Matt wrote:
In case this is important, if I remove the WindowFunction, and only use the FoldFunction it works fine.

I don't see what is wrong...

On Mon, Dec 12, 2016 at 10:53 AM, Matt <[hidden email]> wrote:
Hi,

I'm following the documentation [1] of window functions with incremental aggregations, but I'm getting an "input mismatch" error.

The code [2] is almost identical to the one in the documentation, at the bottom you can find the exact error.

What am I missing? Can you provide a working example of a fold function with both a FoldFunction and a WindowFunction?

Regards,
Matt


[2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
Yassine MARZOUGUI
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Incremental aggregations - Example not working

Yassine MARZOUGUI
Hi Matt,

What version of Flink are you using?
The incremental agregation with fold(ACC, FoldFunction, WindowFunction) in a new change that will be part of Flink 1.2, for Flink 1.1 the correct way to perform incrementation aggregations is : apply(ACC, FoldFunction, WindowFunction) (see the docs for 1.1 [1])


Best,
Yassine

2016-12-12 15:37 GMT+01:00 Chesnay Schepler <[hidden email]>:
Hello Matt,

This looks like a bug in the fold() function to me.

I'm adding Timo to the discussion, he can probably shed some light on this.

Regards,
Chesnay


On 12.12.2016 15:13, Matt wrote:
In case this is important, if I remove the WindowFunction, and only use the FoldFunction it works fine.

I don't see what is wrong...

On Mon, Dec 12, 2016 at 10:53 AM, Matt <[hidden email]> wrote:
Hi,

I'm following the documentation [1] of window functions with incremental aggregations, but I'm getting an "input mismatch" error.

The code [2] is almost identical to the one in the documentation, at the bottom you can find the exact error.

What am I missing? Can you provide a working example of a fold function with both a FoldFunction and a WindowFunction?

Regards,
Matt


[2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
Matt
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Incremental aggregations - Example not working

Matt
I'm using 1.2-SNAPSHOT, should it work in that version?

On Mon, Dec 12, 2016 at 12:18 PM, Yassine MARZOUGUI <[hidden email]> wrote:
Hi Matt,

What version of Flink are you using?
The incremental agregation with fold(ACC, FoldFunction, WindowFunction) in a new change that will be part of Flink 1.2, for Flink 1.1 the correct way to perform incrementation aggregations is : apply(ACC, FoldFunction, WindowFunction) (see the docs for 1.1 [1])


Best,
Yassine

2016-12-12 15:37 GMT+01:00 Chesnay Schepler <[hidden email]>:
Hello Matt,

This looks like a bug in the fold() function to me.

I'm adding Timo to the discussion, he can probably shed some light on this.

Regards,
Chesnay


On 12.12.2016 15:13, Matt wrote:
In case this is important, if I remove the WindowFunction, and only use the FoldFunction it works fine.

I don't see what is wrong...

On Mon, Dec 12, 2016 at 10:53 AM, Matt <[hidden email]> wrote:
Hi,

I'm following the documentation [1] of window functions with incremental aggregations, but I'm getting an "input mismatch" error.

The code [2] is almost identical to the one in the documentation, at the bottom you can find the exact error.

What am I missing? Can you provide a working example of a fold function with both a FoldFunction and a WindowFunction?

Regards,
Matt


[2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
Yassine MARZOUGUI
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Incremental aggregations - Example not working

Yassine MARZOUGUI
Yes, it was suppoed to work. I looked into this, and as Chesnay said, this is a bug in the fold function. I opened an issue in JIRA : https://issues.apache.org/jira/browse/FLINK-5320, and will fix it very soon, thank you for reporting it.
In the mean time you can workaround the problem by specifying the TypeInformation along with the fold function as follows : fold(ACC, FoldFunction, WindowFunction, foldAccumulatorType, resultType). In the example, the foldAccumulatorType is new TupleTypeInfo<Tuple3<String, Long, Integer>>(), and the resultType is also new TupleTypeInfo<Tuple3<String, Long, Integer>>().

Best,
Yassine

2016-12-12 16:38 GMT+01:00 Matt <[hidden email]>:
I'm using 1.2-SNAPSHOT, should it work in that version?

On Mon, Dec 12, 2016 at 12:18 PM, Yassine MARZOUGUI <[hidden email]> wrote:
Hi Matt,

What version of Flink are you using?
The incremental agregation with fold(ACC, FoldFunction, WindowFunction) in a new change that will be part of Flink 1.2, for Flink 1.1 the correct way to perform incrementation aggregations is : apply(ACC, FoldFunction, WindowFunction) (see the docs for 1.1 [1])


Best,
Yassine

2016-12-12 15:37 GMT+01:00 Chesnay Schepler <[hidden email]>:
Hello Matt,

This looks like a bug in the fold() function to me.

I'm adding Timo to the discussion, he can probably shed some light on this.

Regards,
Chesnay


On 12.12.2016 15:13, Matt wrote:
In case this is important, if I remove the WindowFunction, and only use the FoldFunction it works fine.

I don't see what is wrong...

On Mon, Dec 12, 2016 at 10:53 AM, Matt <[hidden email]> wrote:
Hi,

I'm following the documentation [1] of window functions with incremental aggregations, but I'm getting an "input mismatch" error.

The code [2] is almost identical to the one in the documentation, at the bottom you can find the exact error.

What am I missing? Can you provide a working example of a fold function with both a FoldFunction and a WindowFunction?

Regards,
Matt


[2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
Matt
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Incremental aggregations - Example not working

Matt
I just checked with version 1.1.3 and it works fine, the problem is that in that version we can't use Kafka 0.10 if I'm not work. Thank you for the workaround!

Best,
Matt

On Mon, Dec 12, 2016 at 1:52 PM, Yassine MARZOUGUI <[hidden email]> wrote:
Yes, it was suppoed to work. I looked into this, and as Chesnay said, this is a bug in the fold function. I opened an issue in JIRA : https://issues.apache.org/jira/browse/FLINK-5320, and will fix it very soon, thank you for reporting it.
In the mean time you can workaround the problem by specifying the TypeInformation along with the fold function as follows : fold(ACC, FoldFunction, WindowFunction, foldAccumulatorType, resultType). In the example, the foldAccumulatorType is new TupleTypeInfo<Tuple3<String, Long, Integer>>(), and the resultType is also new TupleTypeInfo<Tuple3<String, Long, Integer>>().

Best,
Yassine

2016-12-12 16:38 GMT+01:00 Matt <[hidden email]>:
I'm using 1.2-SNAPSHOT, should it work in that version?

On Mon, Dec 12, 2016 at 12:18 PM, Yassine MARZOUGUI <[hidden email]> wrote:
Hi Matt,

What version of Flink are you using?
The incremental agregation with fold(ACC, FoldFunction, WindowFunction) in a new change that will be part of Flink 1.2, for Flink 1.1 the correct way to perform incrementation aggregations is : apply(ACC, FoldFunction, WindowFunction) (see the docs for 1.1 [1])


Best,
Yassine

2016-12-12 15:37 GMT+01:00 Chesnay Schepler <[hidden email]>:
Hello Matt,

This looks like a bug in the fold() function to me.

I'm adding Timo to the discussion, he can probably shed some light on this.

Regards,
Chesnay


On 12.12.2016 15:13, Matt wrote:
In case this is important, if I remove the WindowFunction, and only use the FoldFunction it works fine.

I don't see what is wrong...

On Mon, Dec 12, 2016 at 10:53 AM, Matt <[hidden email]> wrote:
Hi,

I'm following the documentation [1] of window functions with incremental aggregations, but I'm getting an "input mismatch" error.

The code [2] is almost identical to the one in the documentation, at the bottom you can find the exact error.

What am I missing? Can you provide a working example of a fold function with both a FoldFunction and a WindowFunction?

Regards,
Matt


[2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
Matt
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Incremental aggregations - Example not working

Matt
Err, I meant if I'm not wrong *

On Mon, Dec 12, 2016 at 2:02 PM, Matt <[hidden email]> wrote:
I just checked with version 1.1.3 and it works fine, the problem is that in that version we can't use Kafka 0.10 if I'm not work. Thank you for the workaround!

Best,
Matt

On Mon, Dec 12, 2016 at 1:52 PM, Yassine MARZOUGUI <[hidden email]> wrote:
Yes, it was suppoed to work. I looked into this, and as Chesnay said, this is a bug in the fold function. I opened an issue in JIRA : https://issues.apache.org/jira/browse/FLINK-5320, and will fix it very soon, thank you for reporting it.
In the mean time you can workaround the problem by specifying the TypeInformation along with the fold function as follows : fold(ACC, FoldFunction, WindowFunction, foldAccumulatorType, resultType). In the example, the foldAccumulatorType is new TupleTypeInfo<Tuple3<String, Long, Integer>>(), and the resultType is also new TupleTypeInfo<Tuple3<String, Long, Integer>>().

Best,
Yassine

2016-12-12 16:38 GMT+01:00 Matt <[hidden email]>:
I'm using 1.2-SNAPSHOT, should it work in that version?

On Mon, Dec 12, 2016 at 12:18 PM, Yassine MARZOUGUI <[hidden email]> wrote:
Hi Matt,

What version of Flink are you using?
The incremental agregation with fold(ACC, FoldFunction, WindowFunction) in a new change that will be part of Flink 1.2, for Flink 1.1 the correct way to perform incrementation aggregations is : apply(ACC, FoldFunction, WindowFunction) (see the docs for 1.1 [1])


Best,
Yassine

2016-12-12 15:37 GMT+01:00 Chesnay Schepler <[hidden email]>:
Hello Matt,

This looks like a bug in the fold() function to me.

I'm adding Timo to the discussion, he can probably shed some light on this.

Regards,
Chesnay


On 12.12.2016 15:13, Matt wrote:
In case this is important, if I remove the WindowFunction, and only use the FoldFunction it works fine.

I don't see what is wrong...

On Mon, Dec 12, 2016 at 10:53 AM, Matt <[hidden email]> wrote:
Hi,

I'm following the documentation [1] of window functions with incremental aggregations, but I'm getting an "input mismatch" error.

The code [2] is almost identical to the one in the documentation, at the bottom you can find the exact error.

What am I missing? Can you provide a working example of a fold function with both a FoldFunction and a WindowFunction?

Regards,
Matt


[2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
Free forum by Nabble Edit this page