flink 1.10 support LONG as watermark?

[jingjing bai]

Hi:

flinkers!

I try to upgrade our production to 1.10V from 1.9 which is our current product version.

in our case,the event_time is Long ,and we had implement this function which support long type  as a watermark in our inner version, it is a different from the official version on 1.10.    

on 1.10 version, flink had add watermark definition and I had drop our implement. 

and i encounter this problem too , 

how can I do in new version,  if is ,  I 'm not to migrate our inner implement to new version. 

[Jark Wu-3]

Hi Jingjing,

Event time field must be a TIMESTAMP(3) type. You can convert your Long type value into TIMESTAMP(3) using user-defined function. 

I'm sorry that Flink doesn't provide built-in function for this purpose, but will have one soon. 

For example: 

CREATE TABLE myTable (

 log_ts bigint,

 event_time AS my_func(log_ts),

 WATERMARK FOR event_time AS event_time - INTERVAL '1' SECOND

) WITH (

 ...

);

Here my_func is a UDF which converts BIGINT into TIMESTAMP(3).

Best,

Jark

On Mon, 30 Mar 2020 at 18:16, jingjing bai <[hidden email]> wrote:

Hi:

flinkers!

I try to upgrade our production to 1.10V from 1.9 which is our current product version.

in our case,the event_time is Long ,and we had implement this function which support long type  as a watermark in our inner version, it is a different from the official version on 1.10.    

on 1.10 version, flink had add watermark definition and I had drop our implement. 

and i encounter this problem too , 

how can I do in new version,  if is ,  I 'm not to migrate our inner implement to new version. 

[jingjing bai]

Hi jarkWu!

Is there a FLIP to do so?  I'm very glad to learn from idea.

Best,

jing

Jark Wu <[hidden email]> 于2020年3月30日周一 下午6:52写道：

Hi Jingjing,

Event time field must be a TIMESTAMP(3) type. You can convert your Long type value into TIMESTAMP(3) using user-defined function. 

I'm sorry that Flink doesn't provide built-in function for this purpose, but will have one soon. 

For example: 

CREATE TABLE myTable (

 log_ts bigint,

 event_time AS my_func(log_ts),

 WATERMARK FOR event_time AS event_time - INTERVAL '1' SECOND

) WITH (

 ...

);

Here my_func is a UDF which converts BIGINT into TIMESTAMP(3).

Best,

Jark

On Mon, 30 Mar 2020 at 18:16, jingjing bai <[hidden email]> wrote:

Hi:

flinkers!

I try to upgrade our production to 1.10V from 1.9 which is our current product version.

in our case,the event_time is Long ,and we had implement this function which support long type  as a watermark in our inner version, it is a different from the official version on 1.10.    

on 1.10 version, flink had add watermark definition and I had drop our implement. 

and i encounter this problem too , 

how can I do in new version,  if is ,  I 'm not to migrate our inner implement to new version. 

[Jark Wu-3]

Hi Jing,

I created https://issues.apache.org/jira/browse/FLINK-16889 to support converting from BIGINT to TIMESTAMP. 

Best,

Jark

On Mon, 30 Mar 2020 at 20:30, jingjing bai <[hidden email]> wrote:

Hi jarkWu!

Is there a FLIP to do so?  I'm very glad to learn from idea.

Best,

jing

Jark Wu <[hidden email]> 于2020年3月30日周一 下午6:52写道：

Hi Jingjing,

Event time field must be a TIMESTAMP(3) type. You can convert your Long type value into TIMESTAMP(3) using user-defined function. 

I'm sorry that Flink doesn't provide built-in function for this purpose, but will have one soon. 

For example: 

CREATE TABLE myTable (

 log_ts bigint,

 event_time AS my_func(log_ts),

 WATERMARK FOR event_time AS event_time - INTERVAL '1' SECOND

) WITH (

 ...

);

Here my_func is a UDF which converts BIGINT into TIMESTAMP(3).

Best,

Jark

On Mon, 30 Mar 2020 at 18:16, jingjing bai <[hidden email]> wrote:

Hi:

flinkers!

I try to upgrade our production to 1.10V from 1.9 which is our current product version.

in our case,the event_time is Long ,and we had implement this function which support long type  as a watermark in our inner version, it is a different from the official version on 1.10.    

on 1.10 version, flink had add watermark definition and I had drop our implement. 

and i encounter this problem too , 

how can I do in new version,  if is ,  I 'm not to migrate our inner implement to new version. 

[jingjing bai]

Thanks a lot! 

Jark Wu <[hidden email]> 于2020年4月1日周三 上午1:13写道：

Hi Jing,

I created https://issues.apache.org/jira/browse/FLINK-16889 to support converting from BIGINT to TIMESTAMP. 

Best,

Jark

On Mon, 30 Mar 2020 at 20:30, jingjing bai <[hidden email]> wrote:

Hi jarkWu!

Is there a FLIP to do so?  I'm very glad to learn from idea.

Best,

jing

Jark Wu <[hidden email]> 于2020年3月30日周一 下午6:52写道：

Hi Jingjing,

Event time field must be a TIMESTAMP(3) type. You can convert your Long type value into TIMESTAMP(3) using user-defined function. 

I'm sorry that Flink doesn't provide built-in function for this purpose, but will have one soon. 

For example: 

CREATE TABLE myTable (

 log_ts bigint,

 event_time AS my_func(log_ts),

 WATERMARK FOR event_time AS event_time - INTERVAL '1' SECOND

) WITH (

 ...

);

Here my_func is a UDF which converts BIGINT into TIMESTAMP(3).

Best,

Jark

On Mon, 30 Mar 2020 at 18:16, jingjing bai <[hidden email]> wrote:

Hi:

flinkers!

I try to upgrade our production to 1.10V from 1.9 which is our current product version.

in our case,the event_time is Long ,and we had implement this function which support long type  as a watermark in our inner version, it is a different from the official version on 1.10.    

on 1.10 version, flink had add watermark definition and I had drop our implement. 

and i encounter this problem too , 

how can I do in new version,  if is ,  I 'm not to migrate our inner implement to new version. 

[Jingsong Li]

Hi jingjing,

If seconds precision is OK for you.

You can try "to_timestamp(from_unixtime(your_time_seconds_long))".

Best,

Jingsong Lee

On Wed, Apr 1, 2020 at 8:56 AM jingjing bai <[hidden email]> wrote:

Thanks a lot! 

Jark Wu <[hidden email]> 于2020年4月1日周三 上午1:13写道：

Hi Jing,

I created https://issues.apache.org/jira/browse/FLINK-16889 to support converting from BIGINT to TIMESTAMP. 

Best,

Jark

On Mon, 30 Mar 2020 at 20:30, jingjing bai <[hidden email]> wrote:

Hi jarkWu!

Is there a FLIP to do so?  I'm very glad to learn from idea.

Best,

jing

Jark Wu <[hidden email]> 于2020年3月30日周一 下午6:52写道：

Hi Jingjing,

Event time field must be a TIMESTAMP(3) type. You can convert your Long type value into TIMESTAMP(3) using user-defined function. 

I'm sorry that Flink doesn't provide built-in function for this purpose, but will have one soon. 

For example: 

CREATE TABLE myTable (

 log_ts bigint,

 event_time AS my_func(log_ts),

 WATERMARK FOR event_time AS event_time - INTERVAL '1' SECOND

) WITH (

 ...

);

Here my_func is a UDF which converts BIGINT into TIMESTAMP(3).

Best,

Jark

On Mon, 30 Mar 2020 at 18:16, jingjing bai <[hidden email]> wrote:

Hi:

flinkers!

I try to upgrade our production to 1.10V from 1.9 which is our current product version.

in our case,the event_time is Long ,and we had implement this function which support long type  as a watermark in our inner version, it is a different from the official version on 1.10.    

on 1.10 version, flink had add watermark definition and I had drop our implement. 

and i encounter this problem too , 

how can I do in new version,  if is ,  I 'm not to migrate our inner implement to new version. 

--