Event Timestamp corrupted by timezone

[Wojciech Indyk]

Hi!

I use Flink 1.8 with Scala. I think I've found a problem with event timestamps in TableAPI. When I mark my timestamp: Long as .rowtime and then save it back to stream as sql.Timestamp I will get wrong .getTime result. The gist for reproduction is here: https://gist.github.com/woj-i/b1dfbb71590b7f1c0c58be1f9e41c610

When I change my timezome from GMT+1 to GMT everything works ok.

I've found this post from March http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/event-time-timezone-is-not-correct-tt26457.html but it's not resolved. The most relevant ticket I've found https://issues.apache.org/jira/browse/FLINK-8353 seems to not include the problem I described.

1. Can you confirm it's a bug?

2. Should I post this bug somewhere to be at least planned to solve?

3. Can you recommend me a workaround for the described problem? 

[Lasse Nedergaard]

Hi. 

We have the same Challenges. I asked on Flink forward and it’s a known problem. We input in utc but Flink output in local machine time. We have created a function that converts it back to utc before collecting to down stream. 

Med venlig hilsen / Best regards

Lasse Nedergaard

Den 3. dec. 2019 kl. 15.16 skrev Wojciech Indyk <[hidden email]>:

Hi!

I use Flink 1.8 with Scala. I think I've found a problem with event timestamps in TableAPI. When I mark my timestamp: Long as .rowtime and then save it back to stream as sql.Timestamp I will get wrong .getTime result. The gist for reproduction is here: https://gist.github.com/woj-i/b1dfbb71590b7f1c0c58be1f9e41c610

When I change my timezome from GMT+1 to GMT everything works ok.

I've found this post from March http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/event-time-timezone-is-not-correct-tt26457.html but it's not resolved. The most relevant ticket I've found https://issues.apache.org/jira/browse/FLINK-8353 seems to not include the problem I described.

1. Can you confirm it's a bug?

2. Should I post this bug somewhere to be at least planned to solve?

3. Can you recommend me a workaround for the described problem? 

[Timo Walther]

Hi,

I hope we can solve this issues with the new type system. The core
problem is the old planner uses java.sql.Timestamp which depends on the
timezone of the current machine. I would recommend to set everything to
UTC if possible for now.

Regards,
Timo


On 03.12.19 18:49, Lasse Nedergaard wrote:

> Hi.
>
> We have the same Challenges. I asked on Flink forward and it’s a known
> problem. We input in utc but Flink output in local machine time. We have
> created a function that converts it back to utc before collecting to
> down stream.
>
> Med venlig hilsen / Best regards
> Lasse Nedergaard
>
>
> Den 3. dec. 2019 kl. 15.16 skrev Wojciech Indyk <[hidden email]
> <mailto:[hidden email]>>:
>
>> Hi!
>> I use Flink 1.8 with Scala. I think I've found a problem with event
>> timestamps in TableAPI. When I mark my timestamp: Long as .rowtime and
>> then save it back to stream as sql.Timestamp I will get wrong .getTime
>> result. The gist for reproduction is here:
>> https://gist.github.com/woj-i/b1dfbb71590b7f1c0c58be1f9e41c610
>> When I change my timezome from GMT+1 to GMT everything works ok.
>> I've found this post from March
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/event-time-timezone-is-not-correct-tt26457.html 
>> but it's not resolved. The most relevant ticket I've found
>> https://issues.apache.org/jira/browse/FLINK-8353 seems to not include
>> the problem I described.
>>
>> 1. Can you confirm it's a bug?
>> 2. Should I post this bug somewhere to be at least planned to solve?
>> 3. Can you recommend me a workaround for the described problem?
>>
>> --
>> Kind regards/ Pozdrawiam,
>> Wojciech Indyk

[Jingsong Li]

Hi Wojciech,

You can try 1.9/1.10 with blink planner.

As Timo said, the timestamp is TimestampType without time zone in new type system and Blink planner support it.

And you should use LocalDateTime in sink/down stream, LocalDateTime has no time zone.

Best,

Jingsong Lee

On Tue, Dec 10, 2019 at 11:09 PM Timo Walther <[hidden email]> wrote:

Hi,

I hope we can solve this issues with the new type system. The core
problem is the old planner uses java.sql.Timestamp which depends on the
timezone of the current machine. I would recommend to set everything to
UTC if possible for now.

Regards,
Timo


On 03.12.19 18:49, Lasse Nedergaard wrote:
> Hi.
>
> We have the same Challenges. I asked on Flink forward and it’s a known
> problem. We input in utc but Flink output in local machine time. We have
> created a function that converts it back to utc before collecting to
> down stream.
>
> Med venlig hilsen / Best regards
> Lasse Nedergaard
>
>
> Den 3. dec. 2019 kl. 15.16 skrev Wojciech Indyk <[hidden email]
> <mailto:[hidden email]>>:
>
>> Hi!
>> I use Flink 1.8 with Scala. I think I've found a problem with event
>> timestamps in TableAPI. When I mark my timestamp: Long as .rowtime and
>> then save it back to stream as sql.Timestamp I will get wrong .getTime
>> result. The gist for reproduction is here:
>> https://gist.github.com/woj-i/b1dfbb71590b7f1c0c58be1f9e41c610
>> When I change my timezome from GMT+1 to GMT everything works ok.
>> I've found this post from March
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/event-time-timezone-is-not-correct-tt26457.html
>> but it's not resolved. The most relevant ticket I've found
>> https://issues.apache.org/jira/browse/FLINK-8353 seems to not include
>> the problem I described.
>>
>> 1. Can you confirm it's a bug?
>> 2. Should I post this bug somewhere to be at least planned to solve?
>> 3. Can you recommend me a workaround for the described problem?
>>
>> --
>> Kind regards/ Pozdrawiam,
>> Wojciech Indyk

--