Question about "NoWatermark" in Flink 1.9.2

[Peter Huang]

Hi,

We have a pipeline running while for both datacenters. Since Jan 5th, one of the instances has the issue of  "No WaterMark", In the program, we are using BoundedOutOfdernessTimestampExtractor to get a ts field from each message. The same code runs well in the other dc. We checked the input message, ts field is correctly set. Even we switch auto-watermark, the issue is still there. The graph below shows no Watermark in the second operator. Did anyone experience the same issue before?

Best Regards

Peter Huang

[Peter Huang]

Hi,

We end up finding the root cause. Since a time point, two of the partitions of the input topic don't have any data which causes the second window operator in the pipeline can't receive the watermark of all of the partitions of the first operator. Thus, the watermark can be determined. Hopefully, this finding will be useful for other users in the community. 

Best Regards

Peter Huang

On Thu, Jan 7, 2021 at 8:11 AM Peter Huang <[hidden email]> wrote:

Hi,

We have a pipeline running while for both datacenters. Since Jan 5th, one of the instances has the issue of  "No WaterMark", In the program, we are using BoundedOutOfdernessTimestampExtractor to get a ts field from each message. The same code runs well in the other dc. We checked the input message, ts field is correctly set. Even we switch auto-watermark, the issue is still there. The graph below shows no Watermark in the second operator. Did anyone experience the same issue before?

Best Regards

Peter Huang

[Piotr Nowojski-5]

Hi,

Good to hear that you managed to diagnose the problem. Yes indeed, that's an issue. You can find information on how to deal with idleness here [1].

Best, Piotrek

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/event_timestamps_watermarks.html#dealing-with-idle-sources

pt., 8 sty 2021 o 01:46 Peter Huang <[hidden email]> napisał(a):

Hi,

We end up finding the root cause. Since a time point, two of the partitions of the input topic don't have any data which causes the second window operator in the pipeline can't receive the watermark of all of the partitions of the first operator. Thus, the watermark can be determined. Hopefully, this finding will be useful for other users in the community. 

Best Regards

Peter Huang

On Thu, Jan 7, 2021 at 8:11 AM Peter Huang <[hidden email]> wrote:

Hi,

We have a pipeline running while for both datacenters. Since Jan 5th, one of the instances has the issue of  "No WaterMark", In the program, we are using BoundedOutOfdernessTimestampExtractor to get a ts field from each message. The same code runs well in the other dc. We checked the input message, ts field is correctly set. Even we switch auto-watermark, the issue is still there. The graph below shows no Watermark in the second operator. Did anyone experience the same issue before?

Best Regards

Peter Huang

[Aljoscha Krettek]

Thanks for the update!

Best,
Aljoscha

On 2021/01/07 16:45, Peter Huang wrote:

>Hi,
>
>We end up finding the root cause. Since a time point, two of the partitions
>of the input topic don't have any data which causes the second window
>operator in the pipeline can't receive the watermark of all of the
>partitions of the first operator. Thus, the watermark can be determined.
>Hopefully, this finding will be useful for other users in the community.
>
>
>Best Regards
>Peter Huang
>
>On Thu, Jan 7, 2021 at 8:11 AM Peter Huang <[hidden email]>
>wrote:
>
>> Hi,
>>
>> We have a pipeline running while for both datacenters. Since Jan 5th, one
>> of the instances has the issue of  "No WaterMark", In the program, we are
>> using BoundedOutOfdernessTimestampExtractor to get a ts field from each
>> message. The same code runs well in the other dc. We checked the input
>> message, ts field is correctly set. Even we switch auto-watermark, the
>> issue is still there. The graph below shows no Watermark in the second
>> operator. Did anyone experience the same issue before?
>>
>>
>>
>> [image: image.png]
>>
>>
>> Best Regards
>> Peter Huang
>>