Interval Join Late Record Metrics

[Chris Gillespie]

Hello Flink users, first time poster here.

I'm using an interval join in my Flink project, however I haven't found where late records get logged in metrics. Window Joins have "numLateRecordsDropped" implemented, but is there an equivalent within an interval join? 

My main use case is to track how often a record falls outside of the lower and upper bounds when trying to join two streams. Interval Join looks like it simply short circuits when there is a late record? Maybe I am not understanding what defines a late record in this situation.

Is there a good way to monitor when an interval join fails to join two streams? Currently I'm looking at the delta between two operator metrics, but it hasn't looked that reliable so far.

Thanks,

Chris Gillespie

[Congxian Qiu]

Hi Chris

From the code[1], currently, IntervalJoin will ignore the late data silently, maybe you can create an issue to track this.

[1] https://github.com/apache/flink/blob/5c89d12849ea2aa332126b32808e363f12d436a2/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/IntervalJoinOperator.java#L222

Chris Gillespie <[hidden email]> 于2019年12月11日周三 上午8:09写道：

Hello Flink users, first time poster here.

I'm using an interval join in my Flink project, however I haven't found where late records get logged in metrics. Window Joins have "numLateRecordsDropped" implemented, but is there an equivalent within an interval join? 

My main use case is to track how often a record falls outside of the lower and upper bounds when trying to join two streams. Interval Join looks like it simply short circuits when there is a late record? Maybe I am not understanding what defines a late record in this situation.

Is there a good way to monitor when an interval join fails to join two streams? Currently I'm looking at the delta between two operator metrics, but it hasn't looked that reliable so far.

Thanks,

Chris Gillespie

[Chris Gillespie]

Thanks Congxian, I made a JIRA to track this request. https://issues.apache.org/jira/browse/FLINK-15202

On Wed, Dec 11, 2019 at 12:56 AM Congxian Qiu <[hidden email]> wrote:

Hi Chris

From the code[1], currently, IntervalJoin will ignore the late data silently, maybe you can create an issue to track this.

[1] https://github.com/apache/flink/blob/5c89d12849ea2aa332126b32808e363f12d436a2/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/IntervalJoinOperator.java#L222

Best,

Congxian

Chris Gillespie <[hidden email]> 于2019年12月11日周三 上午8:09写道：

Hello Flink users, first time poster here.

I'm using an interval join in my Flink project, however I haven't found where late records get logged in metrics. Window Joins have "numLateRecordsDropped" implemented, but is there an equivalent within an interval join? 

My main use case is to track how often a record falls outside of the lower and upper bounds when trying to join two streams. Interval Join looks like it simply short circuits when there is a late record? Maybe I am not understanding what defines a late record in this situation.

Is there a good way to monitor when an interval join fails to join two streams? Currently I'm looking at the delta between two operator metrics, but it hasn't looked that reliable so far.

Thanks,

Chris Gillespie