define WATERMARKS in queries/views?

[Őrhidi Mátyás]

Dear Community,

is it possible to define WATERMARKS in SQL queries/views? We have a read only catalog implementation and we would like to assign WMs to the tables somehow.

Thanks,

Matyas

[Jark Wu-3]

Hi Matyas,

You can create a new table based on the existing table using LIKE syntax [1] in the upcoming 1.11 version, e.g. 

CREATE  TABLE derived_table (
    WATERMARK FOR tstmp AS tsmp - INTERVAL '5' SECOND
) LIKE base_table;

For now, maybe you have to manually create a new table using full DDL. 

Best,

Jark

[1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE

2020年4月23日 17:35，Őrhidi Mátyás <[hidden email]> 写道：

Dear Community,

is it possible to define WATERMARKS in SQL queries/views? We have a read only catalog implementation and we would like to assign WMs to the tables somehow.

Thanks,

Matyas

[lec ssmi]

can  assignTimestampAndWatermark  again on a  watermarked table？

Jark Wu <[hidden email]> 于 2020年4月23日周四 20:18写道：

Hi Matyas,

You can create a new table based on the existing table using LIKE syntax [1] in the upcoming 1.11 version, e.g. 

CREATE  TABLE derived_table (
    WATERMARK FOR tstmp AS tsmp - INTERVAL '5' SECOND
) LIKE base_table;

For now, maybe you have to manually create a new table using full DDL. 

Best,

Jark

[1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE

2020年4月23日 17:35，Őrhidi Mátyás <[hidden email]> 写道：

Dear Community,

is it possible to define WATERMARKS in SQL queries/views? We have a read only catalog implementation and we would like to assign WMs to the tables somehow.

Thanks,

Matyas

[Jark Wu-3]

Hi Lec,

You can only define `assignTimestampAndWatermark` on DataStream, so you have to convert `Table` into DataStream first. 

If the DataStream already has been assigned watermark, then the new `assignTimestampAndWatermark` will override the previous watermark. 

Best,

Jark

On Thu, 23 Apr 2020 at 22:03, lec ssmi <[hidden email]> wrote:

can  assignTimestampAndWatermark  again on a  watermarked table？

Jark Wu <[hidden email]> 于 2020年4月23日周四 20:18写道：

Hi Matyas,

You can create a new table based on the existing table using LIKE syntax [1] in the upcoming 1.11 version, e.g. 

CREATE  TABLE derived_table (
    WATERMARK FOR tstmp AS tsmp - INTERVAL '5' SECOND
) LIKE base_table;

For now, maybe you have to manually create a new table using full DDL. 

Best,

Jark

[1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE

2020年4月23日 17:35，Őrhidi Mátyás <[hidden email]> 写道：

Dear Community,

is it possible to define WATERMARKS in SQL queries/views? We have a read only catalog implementation and we would like to assign WMs to the tables somehow.

Thanks,

Matyas

[lec ssmi]

I mean re-assigning watermark and timestamp  in sql queries.  

Jark Wu <[hidden email]> 于2020年4月25日周六 下午9:10写道：

Hi Lec,

You can only define `assignTimestampAndWatermark` on DataStream, so you have to convert `Table` into DataStream first. 

If the DataStream already has been assigned watermark, then the new `assignTimestampAndWatermark` will override the previous watermark. 

Best,

Jark

On Thu, 23 Apr 2020 at 22:03, lec ssmi <[hidden email]> wrote:

can  assignTimestampAndWatermark  again on a  watermarked table？

Jark Wu <[hidden email]> 于 2020年4月23日周四 20:18写道：

Hi Matyas,

You can create a new table based on the existing table using LIKE syntax [1] in the upcoming 1.11 version, e.g. 

CREATE  TABLE derived_table (
    WATERMARK FOR tstmp AS tsmp - INTERVAL '5' SECOND
) LIKE base_table;

For now, maybe you have to manually create a new table using full DDL. 

Best,

Jark

[1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE

2020年4月23日 17:35，Őrhidi Mátyás <[hidden email]> 写道：

Dear Community,

is it possible to define WATERMARKS in SQL queries/views? We have a read only catalog implementation and we would like to assign WMs to the tables somehow.

Thanks,

Matyas

[Jark Wu-3]

Hi Lec,

It's supported in SQL queries. 

Best,

Jark

On Sun, 26 Apr 2020 at 09:10, lec ssmi <[hidden email]> wrote:

I mean re-assigning watermark and timestamp  in sql queries.  

Jark Wu <[hidden email]> 于2020年4月25日周六 下午9:10写道：

Hi Lec,

You can only define `assignTimestampAndWatermark` on DataStream, so you have to convert `Table` into DataStream first. 

If the DataStream already has been assigned watermark, then the new `assignTimestampAndWatermark` will override the previous watermark. 

Best,

Jark

On Thu, 23 Apr 2020 at 22:03, lec ssmi <[hidden email]> wrote:

can  assignTimestampAndWatermark  again on a  watermarked table？

Jark Wu <[hidden email]> 于 2020年4月23日周四 20:18写道：

Hi Matyas,

You can create a new table based on the existing table using LIKE syntax [1] in the upcoming 1.11 version, e.g. 

CREATE  TABLE derived_table (
    WATERMARK FOR tstmp AS tsmp - INTERVAL '5' SECOND
) LIKE base_table;

For now, maybe you have to manually create a new table using full DDL. 

Best,

Jark

[1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE

2020年4月23日 17:35，Őrhidi Mátyás <[hidden email]> 写道：

Dear Community,

is it possible to define WATERMARKS in SQL queries/views? We have a read only catalog implementation and we would like to assign WMs to the tables somehow.

Thanks,

Matyas

[Leonard Xu]

在 2020年4月23日，22:03，lec ssmi <[hidden email]> 写道：

can  assignTimestampAndWatermark  again on a  watermarked table？

Hi, lec

I’m afraid not, multiple watermark statements is not supported yet,

If you need multiple watermarks for one table, you can simply create

multiple tables with different watermark using DDL in you catalog.

Best,

Leonard Xu

Jark Wu <[hidden email]> 于 2020年4月23日周四 20:18写道：

Hi Matyas,

You can create a new table based on the existing table using LIKE syntax [1] in the upcoming 1.11 version, e.g. 

CREATE  TABLE derived_table (
    WATERMARK FOR tstmp AS tsmp - INTERVAL '5' SECOND
) LIKE base_table;

For now, maybe you have to manually create a new table using full DDL. 

Best,

Jark

[1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE

2020年4月23日 17:35，Őrhidi Mátyás <[hidden email]> 写道：

Dear Community,

is it possible to define WATERMARKS in SQL queries/views? We have a read only catalog implementation and we would like to assign WMs to the tables somehow.

Thanks,

Matyas

[lec ssmi]

According to my practice,  it seems that  declaring the timestamp and watermark once again  will not work, sql seems to have no such syntax.

But it's supported in DataStream API.

Leonard Xu <[hidden email]> 于2020年4月26日周日 下午6:30写道：

在 2020年4月23日，22:03，lec ssmi <[hidden email]> 写道：

can  assignTimestampAndWatermark  again on a  watermarked table？

Hi, lec

I’m afraid not, multiple watermark statements is not supported yet,

If you need multiple watermarks for one table, you can simply create

multiple tables with different watermark using DDL in you catalog.

Best,

Leonard Xu

Jark Wu <[hidden email]> 于 2020年4月23日周四 20:18写道：

Hi Matyas,

You can create a new table based on the existing table using LIKE syntax [1] in the upcoming 1.11 version, e.g. 

CREATE  TABLE derived_table (
    WATERMARK FOR tstmp AS tsmp - INTERVAL '5' SECOND
) LIKE base_table;

For now, maybe you have to manually create a new table using full DDL. 

Best,

Jark

[1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE

2020年4月23日 17:35，Őrhidi Mátyás <[hidden email]> 写道：

Dear Community,

is it possible to define WATERMARKS in SQL queries/views? We have a read only catalog implementation and we would like to assign WMs to the tables somehow.

Thanks,

Matyas