Re: Blink Planner Retracting Streams

Posted by John Mathews on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Blink-Planner-Retracting-Streams-tp36024p36071.html

So the difference between Tuple2<Boolean, Row> and CRow is that CRow has a special TypeInformation defined here: https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/runtime/types/CRowTypeInfo.scala#L32 

that returns the TypeInfo of the underlying row, whereas the TypeInformation for Tuple2 will return type information that contains the boolean for the retraction + a nested type info for the row. So all downstream operations that rely on seeing just the row type info now break for us. 

On Wed, Jun 17, 2020 at 9:23 PM Jark Wu <[hidden email]> wrote:

Hi John,

Maybe I misunderstand something, but CRow doesn't have the `getSchema()` method. You can getSchema() on the Table, this also works if you convert the table into Tuple2<Boolean, Row>. 
Actually, there is no big difference between CRow and Tuple2<Boolean, Row>, they both wrap the change flag and the Row. 

Best,

Jark

On Thu, 18 Jun 2020 at 06:39, John Mathews <[hidden email]> wrote:

Hello Godfrey, 

Thanks for the response! 

I think the problem with Tuple2, is that if my understanding is correct of how CRow worked, when CRow's getSchema() was returned it would return the underlying schema of the row it contained. Tuple2 doesn't do that, so it changes/breaks a lot of our downstream code that is relying on the TableSchema to return the underlying row's schema, and not a Tuple schema.

Any thoughts on that issue?

On Wed, Jun 17, 2020 at 12:16 AM godfrey he <[hidden email]> wrote:

hi John,

You can use Tuple2[Boolean, Row] to replace CRow, the StreamTableEnvironment#toRetractStream method return DataStream[(Boolean, T)].

the code looks like:

tEnv.toRetractStream[Row](table).map(new MapFunction[(Boolean, Row), R] {
      override def map(value: (Boolean, Row)): R = ...
    })

Bests,

Godfrey

John Mathews <[hidden email]> 于2020年6月17日周三 下午12:13写道：

Hello,

I am working on migrating from the flink table-planner to the new blink one, and one problem I am running into is that it doesn't seem like Blink has a concept of a CRow, unlike the original table-planner.

I am therefore struggling to figure out how to properly convert a retracting stream to a SingleOutputStreamOperator when using just the Blink planner libraries.

E.g. in the old planner I could do something like this: 

SingleOutputStreamOperator<CRow> stream = tableEnvironment.toRetractStream(table, typeInfo)

                    .map(value -> new CRow(value.f1, value.f0);

but without the CRow, I'm not sure how to accomplish this.

Any suggestions?

Thanks!

John