the group key is retracted

[lec ssmi]

Hi:

  When we use sql for aggregation operation, for example, the following sql

       select count( distinct name) cnt, type  from table  group by type

  Source data can be regarded as bin log data.

  If the type value of a record is updated in the   database, the values before and after the update will be divided into different partitions and handed over to different operators for calculation. Can Retraction happen correctly?

[lec ssmi]

Logically, when retracting , the old value can be sent to the original partition , as long as the group by operator supports that  the two values before and after the update can be  sent to different partitions. But don't know whether  flink sql does this.

刘大龙 <[hidden email]> 于2020年7月1日周三 上午10:23写道：

I think the old value will not retract, because the type value update, it will be calculate in new value, the old value will not be updated

-----原始邮件-----
发件人:"lec ssmi" <[hidden email]>
发送时间:2020-07-01 09:53:39 (星期三)
收件人: flink-user <[hidden email]>
抄送:
主题: the group key is retracted

Hi:

  When we use sql for aggregation operation, for example, the following sql

       select count( distinct name) cnt, type  from table  group by type

  Source data can be regarded as bin log data.

  If the type value of a record is updated in the   database, the values before and after the update will be divided into different partitions and handed over to different operators for calculation. Can Retraction happen correctly?

---

刘大龙

 

浙江大学 控制系 智能系统与控制研究所 工控新楼217

地址：浙江省杭州市浙大路38号浙江大学玉泉校区

Tel：[phone]

[刘大龙]

oh, sorry, if source operator can retract old value, I think it can.

-----原始邮件-----
发件人:"lec ssmi" <[hidden email]>
发送时间:2020-07-01 10:48:38 (星期三)
收件人: "刘大龙" <[hidden email]>, flink-user <[hidden email]>
抄送:
主题: Re: the group key is retracted

Logically, when retracting , the old value can be sent to the original partition , as long as the group by operator supports that  the two values before and after the update can be  sent to different partitions. But don't know whether  flink sql does this.

刘大龙 <[hidden email]> 于2020年7月1日周三 上午10:23写道：

I think the old value will not retract, because the type value update, it will be calculate in new value, the old value will not be updated

-----原始邮件-----
发件人:"lec ssmi" <[hidden email]>
发送时间:2020-07-01 09:53:39 (星期三)
收件人: flink-user <[hidden email]>
抄送:
主题: the group key is retracted

Hi:

  When we use sql for aggregation operation, for example, the following sql

       select count( distinct name) cnt, type  from table  group by type

  Source data can be regarded as bin log data.

  If the type value of a record is updated in the   database, the values before and after the update will be divided into different partitions and handed over to different operators for calculation. Can Retraction happen correctly?

---

刘大龙

 

浙江大学 控制系 智能系统与控制研究所 工控新楼217

地址：浙江省杭州市浙大路38号浙江大学玉泉校区

Tel：[phone]

[Benchao Li-2]

Hi lec ssmi,

>  If the type value of a record is updated in the  database, the values before and after the update will be divided into different partitions and handed over to different operators for calculation.

I think your understanding is correct.

>  Can Retraction happen correctly?

I didn't get your point, can you elaborate your question a little bit?

lec ssmi <[hidden email]> 于2020年7月1日周三 上午9:54写道：

Hi:

  When we use sql for aggregation operation, for example, the following sql

       select count( distinct name) cnt, type  from table  group by type

  Source data can be regarded as bin log data.

  If the type value of a record is updated in the   database, the values before and after the update will be divided into different partitions and handed over to different operators for calculation. Can Retraction happen correctly?

--

[lec ssmi]

The old value is already counted in a partition, and when the above update occurs, will the  count value of the old partition be subtracted by 1, and then added to the new partition?

Benchao Li <[hidden email]> 于2020年7月1日周三 下午1:11写道：

Hi lec ssmi,

>  If the type value of a record is updated in the  database, the values before and after the update will be divided into different partitions and handed over to different operators for calculation.

I think your understanding is correct.

>  Can Retraction happen correctly?

I didn't get your point, can you elaborate your question a little bit?

lec ssmi <[hidden email]> 于2020年7月1日周三 上午9:54写道：

Hi:

  When we use sql for aggregation operation, for example, the following sql

       select count( distinct name) cnt, type  from table  group by type

  Source data can be regarded as bin log data.

  If the type value of a record is updated in the   database, the values before and after the update will be divided into different partitions and handed over to different operators for calculation. Can Retraction happen correctly?

--

Best,
Benchao Li

[Benchao Li-2]

If you are using 1.11 new changelog format, I think it will retract old value from old partition correctly.

If not, (I assume you are using append only changelog) I think it won't retract old value.

lec ssmi <[hidden email]> 于2020年7月1日周三 下午2:39写道：

The old value is already counted in a partition, and when the above update occurs, will the  count value of the old partition be subtracted by 1, and then added to the new partition?

Benchao Li <[hidden email]> 于2020年7月1日周三 下午1:11写道：

Hi lec ssmi,

>  If the type value of a record is updated in the  database, the values before and after the update will be divided into different partitions and handed over to different operators for calculation.

I think your understanding is correct.

>  Can Retraction happen correctly?

I didn't get your point, can you elaborate your question a little bit?

lec ssmi <[hidden email]> 于2020年7月1日周三 上午9:54写道：

Hi:

  When we use sql for aggregation operation, for example, the following sql

       select count( distinct name) cnt, type  from table  group by type

  Source data can be regarded as bin log data.

  If the type value of a record is updated in the   database, the values before and after the update will be divided into different partitions and handed over to different operators for calculation. Can Retraction happen correctly?

--

Best,
Benchao Li

--