Flink stream data ordering/sequence

[Sane Lee]

Hi all,

Do flink have mechanism for dealing with ordered streams? 

As far as I know , flink attaches timestamps to data items once they are processed. What about the timestamps are already in data items? 

For example, if some data item is missing from particular window (according to its ordering/sequence number), how does flink react to it?

Waits or generates missing data according to in-hand data or just ignores  it?

-best

Sana

[Fabian Hueske-2]

Hi Sana,

The feature you are looking for is called event time processing in Flink.
These blog posts should help you to become familiar with the concepts:

1) Event-Time concepts: http://data-artisans.com/how-apache-flink-enables-new-streaming-applications-part-1/
2) Windows in Flink: http://flink.apache.org/news/2015/12/04/Introducing-windows.html
3) Event-Time example use-case: https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana
4) Code for example: https://github.com/dataArtisans/flink-streaming-demo

You can implement several strategies for dealing with late-arriving events.

Best, Fabian

2016-01-29 10:58 GMT+01:00 Sane Lee <[hidden email]>:

Hi all,

Do flink have mechanism for dealing with ordered streams? 

As far as I know , flink attaches timestamps to data items once they are processed. What about the timestamps are already in data items? 

For example, if some data item is missing from particular window (according to its ordering/sequence number), how does flink react to it?

Waits or generates missing data according to in-hand data or just ignores  it?

-best

Sana