How to understand NOW() in SQL when using Table & SQL API to develop a streaming app?

[Longdexin]

From my point of view, the value of NOW() function in SQL is certain by the
time when the streaming app is launched and will not change with the process
time. However, as a new Flink user, I'm not so sure of that. By the way, if
my attemp is to keep the time logic to update all the time, what should I
do?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Till Rohrmann]

Hi Longdexin,

thanks for reaching out to the Flink community. I am pulling in Jark who might be able to help you with this question.

Cheers,

Till

On Thu, Oct 22, 2020 at 2:56 PM Longdexin <[hidden email]> wrote:

From my point of view, the value of NOW() function in SQL is certain by the
time when the streaming app is launched and will not change with the process
time. However, as a new Flink user, I'm not so sure of that. By the way, if
my attemp is to keep the time logic to update all the time, what should I
do?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Jark Wu-3]

Hi Longdexin,

In traditional batch sql, NOW() is executed and determined before the job is submitted and will not change for every processed record.

However, this doesn't make much sense in streaming sql, therefore, NOW() function in Flink is executed for every record. 

Best,

Jark

On Fri, 23 Oct 2020 at 16:30, Till Rohrmann <[hidden email]> wrote:

Hi Longdexin,

thanks for reaching out to the Flink community. I am pulling in Jark who might be able to help you with this question.

Cheers,

Till

On Thu, Oct 22, 2020 at 2:56 PM Longdexin <[hidden email]> wrote:

From my point of view, the value of NOW() function in SQL is certain by the
time when the streaming app is launched and will not change with the process
time. However, as a new Flink user, I'm not so sure of that. By the way, if
my attemp is to keep the time logic to update all the time, what should I
do?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Till Rohrmann]

Quick question Jark: Is this difference in behaviour documented? I couldn't find it in the docs.

Cheers,

Till

On Tue, Oct 27, 2020 at 7:30 AM Jark Wu <[hidden email]> wrote:

Hi Longdexin,

In traditional batch sql, NOW() is executed and determined before the job is submitted and will not change for every processed record.

However, this doesn't make much sense in streaming sql, therefore, NOW() function in Flink is executed for every record. 

Best,

Jark

On Fri, 23 Oct 2020 at 16:30, Till Rohrmann <[hidden email]> wrote:

Hi Longdexin,

thanks for reaching out to the Flink community. I am pulling in Jark who might be able to help you with this question.

Cheers,

Till

On Thu, Oct 22, 2020 at 2:56 PM Longdexin <[hidden email]> wrote:

From my point of view, the value of NOW() function in SQL is certain by the
time when the streaming app is launched and will not change with the process
time. However, as a new Flink user, I'm not so sure of that. By the way, if
my attemp is to keep the time logic to update all the time, what should I
do?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Jark Wu-3]

Hi Till,

The documentation mentions that "this function is not deterministic" where the "not deterministic" means the value of this function is not deterministic for every record.

However, this is not very clear for users. I think we can improve the documentation. 

Best,

Jark

On Tue, 27 Oct 2020 at 15:59, Till Rohrmann <[hidden email]> wrote:

Quick question Jark: Is this difference in behaviour documented? I couldn't find it in the docs.

Cheers,

Till

On Tue, Oct 27, 2020 at 7:30 AM Jark Wu <[hidden email]> wrote:

Hi Longdexin,

In traditional batch sql, NOW() is executed and determined before the job is submitted and will not change for every processed record.

However, this doesn't make much sense in streaming sql, therefore, NOW() function in Flink is executed for every record. 

Best,

Jark

On Fri, 23 Oct 2020 at 16:30, Till Rohrmann <[hidden email]> wrote:

Hi Longdexin,

thanks for reaching out to the Flink community. I am pulling in Jark who might be able to help you with this question.

Cheers,

Till

On Thu, Oct 22, 2020 at 2:56 PM Longdexin <[hidden email]> wrote:

From my point of view, the value of NOW() function in SQL is certain by the
time when the streaming app is launched and will not change with the process
time. However, as a new Flink user, I'm not so sure of that. By the way, if
my attemp is to keep the time logic to update all the time, what should I
do?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Till Rohrmann]

Thanks for the clarification. This improvement would be helpful, I believe.

Cheers,

Till

On Tue, Oct 27, 2020 at 1:19 PM Jark Wu <[hidden email]> wrote:

Hi Till,

The documentation mentions that "this function is not deterministic" where the "not deterministic" means the value of this function is not deterministic for every record.

However, this is not very clear for users. I think we can improve the documentation. 

Best,

Jark

On Tue, 27 Oct 2020 at 15:59, Till Rohrmann <[hidden email]> wrote:

Quick question Jark: Is this difference in behaviour documented? I couldn't find it in the docs.

Cheers,

Till

On Tue, Oct 27, 2020 at 7:30 AM Jark Wu <[hidden email]> wrote:

Hi Longdexin,

In traditional batch sql, NOW() is executed and determined before the job is submitted and will not change for every processed record.

However, this doesn't make much sense in streaming sql, therefore, NOW() function in Flink is executed for every record. 

Best,

Jark

On Fri, 23 Oct 2020 at 16:30, Till Rohrmann <[hidden email]> wrote:

Hi Longdexin,

thanks for reaching out to the Flink community. I am pulling in Jark who might be able to help you with this question.

Cheers,

Till

On Thu, Oct 22, 2020 at 2:56 PM Longdexin <[hidden email]> wrote:

From my point of view, the value of NOW() function in SQL is certain by the
time when the streaming app is launched and will not change with the process
time. However, as a new Flink user, I'm not so sure of that. By the way, if
my attemp is to keep the time logic to update all the time, what should I
do?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Danny Chan-2]

Our behavior also conflicts with the SQL standard, we should also mention this in the document.

Till Rohrmann <[hidden email]> 于2020年10月27日周二 下午10:37写道：

Thanks for the clarification. This improvement would be helpful, I believe.

Cheers,

Till

On Tue, Oct 27, 2020 at 1:19 PM Jark Wu <[hidden email]> wrote:

Hi Till,

The documentation mentions that "this function is not deterministic" where the "not deterministic" means the value of this function is not deterministic for every record.

However, this is not very clear for users. I think we can improve the documentation. 

Best,

Jark

On Tue, 27 Oct 2020 at 15:59, Till Rohrmann <[hidden email]> wrote:

Quick question Jark: Is this difference in behaviour documented? I couldn't find it in the docs.

Cheers,

Till

On Tue, Oct 27, 2020 at 7:30 AM Jark Wu <[hidden email]> wrote:

Hi Longdexin,

In traditional batch sql, NOW() is executed and determined before the job is submitted and will not change for every processed record.

However, this doesn't make much sense in streaming sql, therefore, NOW() function in Flink is executed for every record. 

Best,

Jark

On Fri, 23 Oct 2020 at 16:30, Till Rohrmann <[hidden email]> wrote:

Hi Longdexin,

thanks for reaching out to the Flink community. I am pulling in Jark who might be able to help you with this question.

Cheers,

Till

On Thu, Oct 22, 2020 at 2:56 PM Longdexin <[hidden email]> wrote:

From my point of view, the value of NOW() function in SQL is certain by the
time when the streaming app is launched and will not change with the process
time. However, as a new Flink user, I'm not so sure of that. By the way, if
my attemp is to keep the time logic to update all the time, what should I
do?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Jark Wu-3]

issue created: https://issues.apache.org/jira/browse/FLINK-19861

On Wed, 28 Oct 2020 at 11:00, Danny Chan <[hidden email]> wrote:

Our behavior also conflicts with the SQL standard, we should also mention this in the document.

Till Rohrmann <[hidden email]> 于2020年10月27日周二 下午10:37写道：

Thanks for the clarification. This improvement would be helpful, I believe.

Cheers,

Till

On Tue, Oct 27, 2020 at 1:19 PM Jark Wu <[hidden email]> wrote:

Hi Till,

The documentation mentions that "this function is not deterministic" where the "not deterministic" means the value of this function is not deterministic for every record.

However, this is not very clear for users. I think we can improve the documentation. 

Best,

Jark

On Tue, 27 Oct 2020 at 15:59, Till Rohrmann <[hidden email]> wrote:

Quick question Jark: Is this difference in behaviour documented? I couldn't find it in the docs.

Cheers,

Till

On Tue, Oct 27, 2020 at 7:30 AM Jark Wu <[hidden email]> wrote:

Hi Longdexin,

In traditional batch sql, NOW() is executed and determined before the job is submitted and will not change for every processed record.

However, this doesn't make much sense in streaming sql, therefore, NOW() function in Flink is executed for every record. 

Best,

Jark

On Fri, 23 Oct 2020 at 16:30, Till Rohrmann <[hidden email]> wrote:

Hi Longdexin,

thanks for reaching out to the Flink community. I am pulling in Jark who might be able to help you with this question.

Cheers,

Till

On Thu, Oct 22, 2020 at 2:56 PM Longdexin <[hidden email]> wrote:

From my point of view, the value of NOW() function in SQL is certain by the
time when the streaming app is launched and will not change with the process
time. However, as a new Flink user, I'm not so sure of that. By the way, if
my attemp is to keep the time logic to update all the time, what should I
do?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/