Description of Flink event time processing

[Elias Levy]

The documentation of how Flink handles event time and watermarks is spread across several places.  I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up.

You can find it here: https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing

I'd appreciate feedback, particularly about the correctness of the described behavior.

[Fabian Hueske-2]

Hi Elias,

I agree, the docs lack a coherent discussion of event time features.

Thank you for this write up!

I just skimmed your document and will provide more detailed feedback later.

It would be great to add such a page to the documentation.

Best, Fabian

2018-07-03 3:07 GMT+02:00 Elias Levy <[hidden email]>:

The documentation of how Flink handles event time and watermarks is spread across several places.  I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up.

You can find it here: https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing

I'd appreciate feedback, particularly about the correctness of the described behavior.

[Fabian Hueske-2]

Hi Elias,

Thanks for the great document!

I made a pass over it and left a few comments.

I think we should definitely add this to the documentation.

Thanks,

Fabian

2018-07-04 10:30 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi Elias,

I agree, the docs lack a coherent discussion of event time features.

Thank you for this write up!

I just skimmed your document and will provide more detailed feedback later.

It would be great to add such a page to the documentation.

Best, Fabian

2018-07-03 3:07 GMT+02:00 Elias Levy <[hidden email]>:

The documentation of how Flink handles event time and watermarks is spread across several places.  I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up.

You can find it here: https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing

I'd appreciate feedback, particularly about the correctness of the described behavior.

[Rong Rong]

Hi Elias,

Thanks for putting together the document. This is actually a very good, well-rounded document.

I think you did not to enable access for comments for the link. Would you mind enabling comments for the google doc?

Thanks,

Rong

On Thu, Jul 5, 2018 at 8:39 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the great document!

I made a pass over it and left a few comments.

I think we should definitely add this to the documentation.

Thanks,

Fabian

2018-07-04 10:30 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi Elias,

I agree, the docs lack a coherent discussion of event time features.

Thank you for this write up!

I just skimmed your document and will provide more detailed feedback later.

It would be great to add such a page to the documentation.

Best, Fabian

2018-07-03 3:07 GMT+02:00 Elias Levy <[hidden email]>:

The documentation of how Flink handles event time and watermarks is spread across several places.  I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up.

You can find it here: https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing

I'd appreciate feedback, particularly about the correctness of the described behavior.

[Elias Levy]

Apologies.  Comments are now enabled.

On Thu, Jul 5, 2018 at 6:09 PM Rong Rong <[hidden email]> wrote:

Hi Elias,

Thanks for putting together the document. This is actually a very good, well-rounded document.

I think you did not to enable access for comments for the link. Would you mind enabling comments for the google doc?

Thanks,

Rong

On Thu, Jul 5, 2018 at 8:39 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the great document!

I made a pass over it and left a few comments.

I think we should definitely add this to the documentation.

Thanks,

Fabian

2018-07-04 10:30 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi Elias,

I agree, the docs lack a coherent discussion of event time features.

Thank you for this write up!

I just skimmed your document and will provide more detailed feedback later.

It would be great to add such a page to the documentation.

Best, Fabian

2018-07-03 3:07 GMT+02:00 Elias Levy <[hidden email]>:

The documentation of how Flink handles event time and watermarks is spread across several places.  I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up.

You can find it here: https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing

I'd appreciate feedback, particularly about the correctness of the described behavior.

[Elias Levy]

Thanks for all the comments.  I've updated the document to account for the feedback.  Please take a look.

On Fri, Jul 6, 2018 at 2:33 PM Elias Levy <[hidden email]> wrote:

Apologies.  Comments are now enabled.

On Thu, Jul 5, 2018 at 6:09 PM Rong Rong <[hidden email]> wrote:

Hi Elias,

Thanks for putting together the document. This is actually a very good, well-rounded document.

I think you did not to enable access for comments for the link. Would you mind enabling comments for the google doc?

Thanks,

Rong

On Thu, Jul 5, 2018 at 8:39 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the great document!

I made a pass over it and left a few comments.

I think we should definitely add this to the documentation.

Thanks,

Fabian

2018-07-04 10:30 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi Elias,

I agree, the docs lack a coherent discussion of event time features.

Thank you for this write up!

I just skimmed your document and will provide more detailed feedback later.

It would be great to add such a page to the documentation.

Best, Fabian

2018-07-03 3:07 GMT+02:00 Elias Levy <[hidden email]>:

The documentation of how Flink handles event time and watermarks is spread across several places.  I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up.

You can find it here: https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing

I'd appreciate feedback, particularly about the correctness of the described behavior.

[Fabian Hueske-2]

Hi Elias,

Thanks for the update!

I'll try to have another look soon. 

Best, Fabian

2018-07-11 1:30 GMT+02:00 Elias Levy <[hidden email]>:

Thanks for all the comments.  I've updated the document to account for the feedback.  Please take a look.

On Fri, Jul 6, 2018 at 2:33 PM Elias Levy <[hidden email]> wrote:

Apologies.  Comments are now enabled.

On Thu, Jul 5, 2018 at 6:09 PM Rong Rong <[hidden email]> wrote:

Hi Elias,

Thanks for putting together the document. This is actually a very good, well-rounded document.

I think you did not to enable access for comments for the link. Would you mind enabling comments for the google doc?

Thanks,

Rong

On Thu, Jul 5, 2018 at 8:39 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the great document!

I made a pass over it and left a few comments.

I think we should definitely add this to the documentation.

Thanks,

Fabian

2018-07-04 10:30 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi Elias,

I agree, the docs lack a coherent discussion of event time features.

Thank you for this write up!

I just skimmed your document and will provide more detailed feedback later.

It would be great to add such a page to the documentation.

Best, Fabian

2018-07-03 3:07 GMT+02:00 Elias Levy <[hidden email]>:

The documentation of how Flink handles event time and watermarks is spread across several places.  I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up.

You can find it here: https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing

I'd appreciate feedback, particularly about the correctness of the described behavior.

[Elias Levy]

Fabian,

You have any time to review the changes?

On Thu, Jul 19, 2018 at 2:19 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the update!

I'll try to have another look soon. 

Best, Fabian

2018-07-11 1:30 GMT+02:00 Elias Levy <[hidden email]>:

Thanks for all the comments.  I've updated the document to account for the feedback.  Please take a look.

On Fri, Jul 6, 2018 at 2:33 PM Elias Levy <[hidden email]> wrote:

Apologies.  Comments are now enabled.

On Thu, Jul 5, 2018 at 6:09 PM Rong Rong <[hidden email]> wrote:

Hi Elias,

Thanks for putting together the document. This is actually a very good, well-rounded document.

I think you did not to enable access for comments for the link. Would you mind enabling comments for the google doc?

Thanks,

Rong

On Thu, Jul 5, 2018 at 8:39 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the great document!

I made a pass over it and left a few comments.

I think we should definitely add this to the documentation.

Thanks,

Fabian

2018-07-04 10:30 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi Elias,

I agree, the docs lack a coherent discussion of event time features.

Thank you for this write up!

I just skimmed your document and will provide more detailed feedback later.

It would be great to add such a page to the documentation.

Best, Fabian

2018-07-03 3:07 GMT+02:00 Elias Levy <[hidden email]>:

The documentation of how Flink handles event time and watermarks is spread across several places.  I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up.

You can find it here: https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing

I'd appreciate feedback, particularly about the correctness of the described behavior.

[Fabian Hueske-2]

Hi Elias,

Sorry for the delay. I just made a pass over the document.

I think it is very good.

Let's have a look where this fits best into the docs and check if there is duplicate content on other pages that should be removed / reorganized.

Best, Fabian

2018-07-31 3:17 GMT+02:00 Elias Levy <[hidden email]>:

Fabian,

You have any time to review the changes?

On Thu, Jul 19, 2018 at 2:19 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the update!

I'll try to have another look soon. 

Best, Fabian

2018-07-11 1:30 GMT+02:00 Elias Levy <[hidden email]>:

Thanks for all the comments.  I've updated the document to account for the feedback.  Please take a look.

On Fri, Jul 6, 2018 at 2:33 PM Elias Levy <[hidden email]> wrote:

Apologies.  Comments are now enabled.

On Thu, Jul 5, 2018 at 6:09 PM Rong Rong <[hidden email]> wrote:

Hi Elias,

Thanks for putting together the document. This is actually a very good, well-rounded document.

I think you did not to enable access for comments for the link. Would you mind enabling comments for the google doc?

Thanks,

Rong

On Thu, Jul 5, 2018 at 8:39 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the great document!

I made a pass over it and left a few comments.

I think we should definitely add this to the documentation.

Thanks,

Fabian

2018-07-04 10:30 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi Elias,

I agree, the docs lack a coherent discussion of event time features.

Thank you for this write up!

I just skimmed your document and will provide more detailed feedback later.

It would be great to add such a page to the documentation.

Best, Fabian

2018-07-03 3:07 GMT+02:00 Elias Levy <[hidden email]>:

The documentation of how Flink handles event time and watermarks is spread across several places.  I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up.

You can find it here: https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing

I'd appreciate feedback, particularly about the correctness of the described behavior.

[Fabian Hueske-2]

Hi Elias,

Thanks for the update!

I think the document can be added to the docs now.

It has some overlap with the Event Time Overview page.

IMO, it should not replace the overview page but rather be a new page. Maybe, we can make the overview a bit slimmer and point to the more detailed discussion of your document.

Elias, do you want to put your document into Markdown and open a PR for the documentation?

Thanks,

Fabian

2018-07-31 18:16 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi Elias,

Sorry for the delay. I just made a pass over the document.

I think it is very good.

Let's have a look where this fits best into the docs and check if there is duplicate content on other pages that should be removed / reorganized.

Best, Fabian

2018-07-31 3:17 GMT+02:00 Elias Levy <[hidden email]>:

Fabian,

You have any time to review the changes?

On Thu, Jul 19, 2018 at 2:19 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the update!

I'll try to have another look soon. 

Best, Fabian

2018-07-11 1:30 GMT+02:00 Elias Levy <[hidden email]>:

Thanks for all the comments.  I've updated the document to account for the feedback.  Please take a look.

On Fri, Jul 6, 2018 at 2:33 PM Elias Levy <[hidden email]> wrote:

Apologies.  Comments are now enabled.

On Thu, Jul 5, 2018 at 6:09 PM Rong Rong <[hidden email]> wrote:

Hi Elias,

Thanks for putting together the document. This is actually a very good, well-rounded document.

I think you did not to enable access for comments for the link. Would you mind enabling comments for the google doc?

Thanks,

Rong

On Thu, Jul 5, 2018 at 8:39 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the great document!

I made a pass over it and left a few comments.

I think we should definitely add this to the documentation.

Thanks,

Fabian

2018-07-04 10:30 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi Elias,

I agree, the docs lack a coherent discussion of event time features.

Thank you for this write up!

I just skimmed your document and will provide more detailed feedback later.

It would be great to add such a page to the documentation.

Best, Fabian

2018-07-03 3:07 GMT+02:00 Elias Levy <[hidden email]>:

The documentation of how Flink handles event time and watermarks is spread across several places.  I've been wanting a single location that summarizes the subject, and as none was available, I wrote one up.

You can find it here: https://docs.google.com/document/d/1b5d-hTdJQsPH3YD0zTB4ZqodinZVHFomKvt41FfUPMc/edit?usp=sharing

I'd appreciate feedback, particularly about the correctness of the described behavior.

[Elias Levy]

Fabian,

https://github.com/apache/flink/pull/6481

I added a page, but did not remove or edit any existing page.  Let me know what you'd like to see trimmed.

On Thu, Aug 2, 2018 at 8:44 AM Fabian Hueske <[hidden email]> wrote:

Hi Elias,

Thanks for the update!

I think the document can be added to the docs now.

It has some overlap with the Event Time Overview page.

IMO, it should not replace the overview page but rather be a new page. Maybe, we can make the overview a bit slimmer and point to the more detailed discussion of your document.

Elias, do you want to put your document into Markdown and open a PR for the documentation?