Ingesting a table from a retraction stream !!

[Puneet Kinra-2]

Hi 

I know currently Ingesting a table from a retraction stream is not supported yet. 

is there any plan to include in upcoming releases.

--

[Timo Walther]

Hi Puneet,

I'm very sure that this feature will be included in upcoming releases. However, such an interface has to be carefully designed because built-in operators assume that only records that have been emitted previously are retracted. Maybe it will be part of Flink 1.6 but 1.7 should definitely be possible.

As a workaround you could implement a custom rule that translates parts of your plan into a custom DataStream operator. This might require some investigation how the translation is done internally because this is not documented. I don't know if it would be worth the effort. You might take a look at TableEnvironment.getConfig().setCalciteConfig() where you can add additional rules. You can use the available rules in org.apache.flink.table.plan.rules.FlinkRuleSets as a reference.

I hope that helps.

Regards,
Timo



Am 1/26/18 um 4:08 PM schrieb Puneet Kinra:

Hi 

I know currently Ingesting a table from a retraction stream is not supported yet. 

is there any plan to include in upcoming releases.

--

[Hequn Cheng]

Hi Puneet, 

I agree with Timo, such a feature should be carefully designed.

Besides, I'm very interested with your business scenario. I am wondering where your retraction stream come from (Maybe mysql binlog? or ?).

It would be great if you can tell more about the details. Maybe we can figure out a simple workaround.

Thanks, Hequn.

2018-01-26 23:08 GMT+08:00 Puneet Kinra <[hidden email]>:

Hi 

I know currently Ingesting a table from a retraction stream is not supported yet. 

is there any plan to include in upcoming releases.

--

[Puneet Kinra-2]

Hi Hequn 

Actually we are building a offer management system or real time contextual marketing platform. I am creating a poc as of now

over initial design is some thing like this.

e.g ---

1)  2 Sources are registered as as Streaming Table Source.

         a) Assume A with Table Name --TestA (append able mode).

         b) Assume B with Table Name --TestB (append able mode).

2) I have set of predefined aggregators  that run on both the tables A and B

         a) Assume agg1 run on table TestA ;

         b) Assume agg2 run on table TestB ;

3) Now I want to store these aggregators in third table along maintaining the "UID" (unique key)

    not append able mode  then want to run the selection criteria 

    

     for instance: select * from TableC where agg1 >0 and agg2>20;

On Sat, Jan 27, 2018 at 10:27 AM, Hequn Cheng <[hidden email]> wrote:

Hi Puneet, 

I agree with Timo, such a feature should be carefully designed.

Besides, I'm very interested with your business scenario. I am wondering where your retraction stream come from (Maybe mysql binlog? or ?).

It would be great if you can tell more about the details. Maybe we can figure out a simple workaround.

Thanks, Hequn.

2018-01-26 23:08 GMT+08:00 Puneet Kinra <[hidden email]>:

Hi 

I know currently Ingesting a table from a retraction stream is not supported yet. 

is there any plan to include in upcoming releases.

--

--

[Hequn Cheng]

Hi Puneet, 

OK, thanks for your information. 

"a) Assume agg1 run on table TestA ;"

If there is only one group by between source and sink, you can ingest the output agg result with append mode then followed with a group by to translate it into update mode(group by uid and do LAST_VALUE agg). 

If there are two or more group by between source and sink, this workaround is not working, because two level group by may output delete messages and the corresponding key of the delete message may not update again, i.e, the key may be deleted forever.

 "b) Assume agg2 run on table TestB ;"

same with a)

2018-01-27 16:09 GMT+08:00 Puneet Kinra <[hidden email]>:

Hi Hequn 

Actually we are building a offer management system or real time contextual marketing platform. I am creating a poc as of now

over initial design is some thing like this.

e.g ---

1)  2 Sources are registered as as Streaming Table Source.

         a) Assume A with Table Name --TestA (append able mode).

         b) Assume B with Table Name --TestB (append able mode).

2) I have set of predefined aggregators  that run on both the tables A and B

         a) Assume agg1 run on table TestA ;

         b) Assume agg2 run on table TestB ;

3) Now I want to store these aggregators in third table along maintaining the "UID" (unique key)

    not append able mode  then want to run the selection criteria 

    

     for instance: select * from TableC where agg1 >0 and agg2>20;

On Sat, Jan 27, 2018 at 10:27 AM, Hequn Cheng <[hidden email]> wrote:

Hi Puneet, 

I agree with Timo, such a feature should be carefully designed.

Besides, I'm very interested with your business scenario. I am wondering where your retraction stream come from (Maybe mysql binlog? or ?).

It would be great if you can tell more about the details. Maybe we can figure out a simple workaround.

Thanks, Hequn.

2018-01-26 23:08 GMT+08:00 Puneet Kinra <[hidden email]>:

Hi 

I know currently Ingesting a table from a retraction stream is not supported yet. 

is there any plan to include in upcoming releases.

--

--