checkpoint delay consume message

[nick toker-2]

Hello,

We noticed the following behavior:
If we enable the flink checkpoints, we saw that there is a delay between the time we write a message to the KAFKA topic and the time the flink kafka connector consumes this message.
The delay is closely related to checkpointInterval and/or minPauseBetweenCheckpoints meening that the MAX delay when consuming a message from KAFKA will be one of these parameters.

Could you please advise how we can remove/control this delay?

we use flink 1.11.2

BR

nick

[Yun Gao]

 Hi Nick,

    Are you using EXACTLY_ONCE semantics ? If so the sink would use transactions, and only commit the transaction on checkpoint complete to ensure end-to-end exactly-once. A detailed description could be find in [1]

Best,

 Yun

[1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html

------------------------------------------------------------------
Sender:nick toker<[hidden email]>
Date:2020/12/21 23:52:34
Recipient:user<[hidden email]>
Theme:checkpoint delay consume message

Hello,

We noticed the following behavior:
If we enable the flink checkpoints, we saw that there is a delay between the time we write a message to the KAFKA topic and the time the flink kafka connector consumes this message.
The delay is closely related to checkpointInterval and/or minPauseBetweenCheckpoints meening that the MAX delay when consuming a message from KAFKA will be one of these parameters.

Could you please advise how we can remove/control this delay?

we use flink 1.11.2

BR

nick

[nick toker-2]

hi

i am confused

the delay in in the source when reading message not on the sink

nick

‫בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת ‪Yun Gao‬‏ <‪[hidden email]‬‏>:‬

 Hi Nick,

    Are you using EXACTLY_ONCE semantics ? If so the sink would use transactions, and only commit the transaction on checkpoint complete to ensure end-to-end exactly-once. A detailed description could be find in [1]

Best,

 Yun

[1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html

------------------------------------------------------------------
Sender:nick toker<[hidden email]>
Date:2020/12/21 23:52:34
Recipient:user<[hidden email]>
Theme:checkpoint delay consume message

Hello,

We noticed the following behavior:
If we enable the flink checkpoints, we saw that there is a delay between the time we write a message to the KAFKA topic and the time the flink kafka connector consumes this message.
The delay is closely related to checkpointInterval and/or minPauseBetweenCheckpoints meening that the MAX delay when consuming a message from KAFKA will be one of these parameters.

Could you please advise how we can remove/control this delay?

we use flink 1.11.2

BR

nick

[Yun Gao]

Hi nick,

   Sorry I initially think that the data is also write into Kafka with flink . So it could be ensured that there is no delay in the write side, right ? Does the delay in the read side keeps existing ?

Best,

 Yun

------------------Original Mail ------------------

Sender:nick toker <[hidden email]>

Send Date:Tue Dec 22 01:43:50 2020

Recipients:Yun Gao <[hidden email]>

CC:user <[hidden email]>

Subject:Re: checkpoint delay consume message

hi

i am confused

the delay in in the source when reading message not on the sink

nick

‫בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת ‪Yun Gao‬‏ <‪[hidden email]‬‏>:‬

 Hi Nick,

    Are you using EXACTLY_ONCE semantics ? If so the sink would use transactions, and only commit the transaction on checkpoint complete to ensure end-to-end exactly-once. A detailed description could be find in [1]

Best,

 Yun

[1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html

------------------------------------------------------------------
Sender:nick toker<[hidden email]>
Date:2020/12/21 23:52:34
Recipient:user<[hidden email]>
Theme:checkpoint delay consume message

Hello,

We noticed the following behavior:
If we enable the flink checkpoints, we saw that there is a delay between the time we write a message to the KAFKA topic and the time the flink kafka connector consumes this message.
The delay is closely related to checkpointInterval and/or minPauseBetweenCheckpoints meening that the MAX delay when consuming a message from KAFKA will be one of these parameters.

Could you please advise how we can remove/control this delay?

we use flink 1.11.2

BR

nick

[nick toker-2]

Hi Yun,

Sorry but we didn't understand your questions.

The delay we are experiencing is on the read side. 

The message is written to kafka topic and consumed by flink with a delay that depends on the checkpoints interval

When we disabled the checkpoints the messages are immediately consumed

We use the EXACTLY-ONCE semantic.

Please advise.

BR,

Nick

‫בתאריך יום ג׳, 22 בדצמ׳ 2020 ב-9:32 מאת ‪Yun Gao‬‏ <‪[hidden email]‬‏>:‬

Hi nick,

   Sorry I initially think that the data is also write into Kafka with flink . So it could be ensured that there is no delay in the write side, right ? Does the delay in the read side keeps existing ?

Best,

 Yun

------------------Original Mail ------------------

Sender:nick toker <[hidden email]>

Send Date:Tue Dec 22 01:43:50 2020

Recipients:Yun Gao <[hidden email]>

CC:user <[hidden email]>

Subject:Re: checkpoint delay consume message

hi

i am confused

the delay in in the source when reading message not on the sink

nick

‫בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת ‪Yun Gao‬‏ <‪[hidden email]‬‏>:‬

 Hi Nick,

    Are you using EXACTLY_ONCE semantics ? If so the sink would use transactions, and only commit the transaction on checkpoint complete to ensure end-to-end exactly-once. A detailed description could be find in [1]

Best,

 Yun

[1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html

------------------------------------------------------------------
Sender:nick toker<[hidden email]>
Date:2020/12/21 23:52:34
Recipient:user<[hidden email]>
Theme:checkpoint delay consume message

Hello,

We noticed the following behavior:
If we enable the flink checkpoints, we saw that there is a delay between the time we write a message to the KAFKA topic and the time the flink kafka connector consumes this message.
The delay is closely related to checkpointInterval and/or minPauseBetweenCheckpoints meening that the MAX delay when consuming a message from KAFKA will be one of these parameters.

Could you please advise how we can remove/control this delay?

we use flink 1.11.2

BR

nick

[lec ssmi]

Checkpoint can be done synchronously and  asynchronously,  the latter is the default .

If you chooese  the synchronous way , it may cause this problem.

nick toker <[hidden email]> 于2020年12月23日周三 下午3:53写道：

Hi Yun,

Sorry but we didn't understand your questions.

The delay we are experiencing is on the read side. 

The message is written to kafka topic and consumed by flink with a delay that depends on the checkpoints interval

When we disabled the checkpoints the messages are immediately consumed

We use the EXACTLY-ONCE semantic.

Please advise.

BR,

Nick

‫בתאריך יום ג׳, 22 בדצמ׳ 2020 ב-9:32 מאת ‪Yun Gao‬‏ <‪[hidden email]‬‏>:‬

Hi nick,

   Sorry I initially think that the data is also write into Kafka with flink . So it could be ensured that there is no delay in the write side, right ? Does the delay in the read side keeps existing ?

Best,

 Yun

------------------Original Mail ------------------

Sender:nick toker <[hidden email]>

Send Date:Tue Dec 22 01:43:50 2020

Recipients:Yun Gao <[hidden email]>

CC:user <[hidden email]>

Subject:Re: checkpoint delay consume message

hi

i am confused

the delay in in the source when reading message not on the sink

nick

‫בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת ‪Yun Gao‬‏ <‪[hidden email]‬‏>:‬

 Hi Nick,

    Are you using EXACTLY_ONCE semantics ? If so the sink would use transactions, and only commit the transaction on checkpoint complete to ensure end-to-end exactly-once. A detailed description could be find in [1]

Best,

 Yun

[1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html

------------------------------------------------------------------
Sender:nick toker<[hidden email]>
Date:2020/12/21 23:52:34
Recipient:user<[hidden email]>
Theme:checkpoint delay consume message

Hello,

We noticed the following behavior:
If we enable the flink checkpoints, we saw that there is a delay between the time we write a message to the KAFKA topic and the time the flink kafka connector consumes this message.
The delay is closely related to checkpointInterval and/or minPauseBetweenCheckpoints meening that the MAX delay when consuming a message from KAFKA will be one of these parameters.

Could you please advise how we can remove/control this delay?

we use flink 1.11.2

BR

nick

[nick toker-2]

Hi,
Hi,  We think  we are using the default values unless we are missing something.
So this doesn't explain the problem we are facing.
Could you please tell us how to choose synchronous or asynchronous checkpoints just to be sure we are using the correct configuration ?
BR,
Nick

‫בתאריך יום ה׳, 24 בדצמ׳ 2020 ב-3:36 מאת ‪lec ssmi‬‏ <‪[hidden email]‬‏>:‬

Checkpoint can be done synchronously and  asynchronously,  the latter is the default .

If you chooese  the synchronous way , it may cause this problem.

nick toker <[hidden email]> 于2020年12月23日周三 下午3:53写道：

Hi Yun,

Sorry but we didn't understand your questions.

The delay we are experiencing is on the read side. 

The message is written to kafka topic and consumed by flink with a delay that depends on the checkpoints interval

When we disabled the checkpoints the messages are immediately consumed

We use the EXACTLY-ONCE semantic.

Please advise.

BR,

Nick

‫בתאריך יום ג׳, 22 בדצמ׳ 2020 ב-9:32 מאת ‪Yun Gao‬‏ <‪[hidden email]‬‏>:‬

Hi nick,

   Sorry I initially think that the data is also write into Kafka with flink . So it could be ensured that there is no delay in the write side, right ? Does the delay in the read side keeps existing ?

Best,

 Yun

------------------Original Mail ------------------

Sender:nick toker <[hidden email]>

Send Date:Tue Dec 22 01:43:50 2020

Recipients:Yun Gao <[hidden email]>

CC:user <[hidden email]>

Subject:Re: checkpoint delay consume message

hi

i am confused

the delay in in the source when reading message not on the sink

nick

‫בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת ‪Yun Gao‬‏ <‪[hidden email]‬‏>:‬

 Hi Nick,

    Are you using EXACTLY_ONCE semantics ? If so the sink would use transactions, and only commit the transaction on checkpoint complete to ensure end-to-end exactly-once. A detailed description could be find in [1]

Best,

 Yun

[1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html

------------------------------------------------------------------
Sender:nick toker<[hidden email]>
Date:2020/12/21 23:52:34
Recipient:user<[hidden email]>
Theme:checkpoint delay consume message

Hello,

We noticed the following behavior:
If we enable the flink checkpoints, we saw that there is a delay between the time we write a message to the KAFKA topic and the time the flink kafka connector consumes this message.
The delay is closely related to checkpointInterval and/or minPauseBetweenCheckpoints meening that the MAX delay when consuming a message from KAFKA will be one of these parameters.

Could you please advise how we can remove/control this delay?

we use flink 1.11.2

BR

nick

[Arvid Heise-3]

There seems to be a double-post with the mail "Long latency when consuming a message from KAFKA and checkpoint is enabled". Let's continue discussion there.

On Sun, Dec 27, 2020 at 8:33 AM nick toker <[hidden email]> wrote:

Hi,
Hi,  We think  we are using the default values unless we are missing something.
So this doesn't explain the problem we are facing.
Could you please tell us how to choose synchronous or asynchronous checkpoints just to be sure we are using the correct configuration ?
BR,
Nick

‫בתאריך יום ה׳, 24 בדצמ׳ 2020 ב-3:36 מאת ‪lec ssmi‬‏ <‪[hidden email]‬‏>:‬

Checkpoint can be done synchronously and  asynchronously,  the latter is the default .

If you chooese  the synchronous way , it may cause this problem.

nick toker <[hidden email]> 于2020年12月23日周三 下午3:53写道：

Hi Yun,

Sorry but we didn't understand your questions.

The delay we are experiencing is on the read side. 

The message is written to kafka topic and consumed by flink with a delay that depends on the checkpoints interval

When we disabled the checkpoints the messages are immediately consumed

We use the EXACTLY-ONCE semantic.

Please advise.

BR,

Nick

‫בתאריך יום ג׳, 22 בדצמ׳ 2020 ב-9:32 מאת ‪Yun Gao‬‏ <‪[hidden email]‬‏>:‬

Hi nick,

   Sorry I initially think that the data is also write into Kafka with flink . So it could be ensured that there is no delay in the write side, right ? Does the delay in the read side keeps existing ?

Best,

 Yun

------------------Original Mail ------------------

Sender:nick toker <[hidden email]>

Send Date:Tue Dec 22 01:43:50 2020

Recipients:Yun Gao <[hidden email]>

CC:user <[hidden email]>

Subject:Re: checkpoint delay consume message

hi

i am confused

the delay in in the source when reading message not on the sink

nick

‫בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת ‪Yun Gao‬‏ <‪[hidden email]‬‏>:‬

 Hi Nick,

    Are you using EXACTLY_ONCE semantics ? If so the sink would use transactions, and only commit the transaction on checkpoint complete to ensure end-to-end exactly-once. A detailed description could be find in [1]

Best,

 Yun

[1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html

------------------------------------------------------------------
Sender:nick toker<[hidden email]>
Date:2020/12/21 23:52:34
Recipient:user<[hidden email]>
Theme:checkpoint delay consume message

Hello,

We noticed the following behavior:
If we enable the flink checkpoints, we saw that there is a delay between the time we write a message to the KAFKA topic and the time the flink kafka connector consumes this message.
The delay is closely related to checkpointInterval and/or minPauseBetweenCheckpoints meening that the MAX delay when consuming a message from KAFKA will be one of these parameters.

Could you please advise how we can remove/control this delay?

we use flink 1.11.2

BR

nick

--