[ANNOUNCE] Weekly Community Update 2019/27

[Konstantin Knauf-2]

Dear community, 

this week's community digest with news on Flink 1.9.0 and Flink 1.8.1, our Travis setup, Flink on PyPi, and a couple of new initiatives around the DataStream API.

As always, please feel free to add additional updates and news to this thread!

Flink Development

===============

* [releases] The feature freeze for Flink 1.9.0 has been announced on Friday. The release branch will only be cut next Thursday though to allow for some finishing strokes mainly on the planned features for the Table API. [1]

* [releases] Flink 1.8.1 has been released. The release blog post [2] contains a list of all resolved issues.

* [development process] Over the last weeks there has been an ongoing discussion on how to deal with the lack of resources available for Flink's CI (Travis). This week Chesnay has started a voting thread to move away from ASF's Travis account to a Travis account sponsored by Ververica. This got a lot of approvals, but the actual implementation is not trivial as we can not link an external Travis account to the apache/flink repository. The approach currently poc'ed by Chensay involves us fetching every PR to a ververica-controlled repository and publishing the results back to apache/flink. [3]

* [python] Flink 1.9.0 will contain a first version of a Python Table API. Consequently, Jincheng started a discussion to publish it to PyPi. The name will simply be "apache-flink". Currently, the main question is whether the python package should bundle the binary distribution of Apache Flink (Java/Scala) it depends on. It seems more feedback is needed to move this forward. [4]

* [datastream-api] With FLIP-34 the Flink community introduced a the "stop" command for graceful shutdown of a Flink Job. UDFs currently only have a close() method, which is called for any type of shutdown. Now, Klou started a discussion about adding an optional interface, which methods would only be called during gracelful shutdown scenarios (but e.g. not in the case of failure). [5,6]

* [datastream-api] Xingcan has started a VOTE on how to proceed with the split/select API. Voting Time ended one ago. More opinions wouldn't hurt to move this forward. [7]

* [state management] After an offline sync between Vino and Kurt the discussion on "local keyed state" and "local aggregations" covered last week will now first focus on the proposal of "local keyed state" and "localKeyBy()" as this was the core of Vino's original proposal.  [8]

* [logging] An old discussion thread [9] by Vino on separating the logs between Flink Jobs within the same cluster was revived a bit this week in the context of a new ticket [10]. A lot of the (interesting) discussion has moved to this ticket for now.

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Feature-freeze-for-Apache-Flink-1-9-0-release-td29751.html

[2] https://flink.apache.org/news/2019/07/02/release-1.8.1.html

[3] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-solve-unstable-build-capacity-problem-on-TravisCI-tp29881.html

[4] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Publish-the-PyFlink-into-PyPI-td30095.html

[5] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Graceful-Shutdown-Handling-by-UDFs-td30229.html

[6]https://docs.google.com/document/d/1SXfhmeiJfWqi2ITYgCgAoSDUv5PNq1T8Zu01nR5Ebog/edit

[7] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-How-to-Deal-with-Split-Select-in-DataStream-API-td30233.html

[8] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Local-Aggregation-in-Flink-td29307.html

[9] https://issues.apache.org/jira/browse/FLINK-12953

[10] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-framework-and-user-log-separation-td27441.html

Notable Bugs

===========

* [FLINK-13063] [1.8.1] [1.7.2] [1.6.4] In an operator chain containing an AsyncOperator records can be lost or duplicated due to a bug in the AsyncOperator. Until the mailbox model has been fully implemented, the fix ensures that the AsyncOperator can only be the head of an operator. Fixed in 1.7.3 and 1.8.2. [11]

* [FLINK-12889] [1.8.1] [1.7.2] In the case where a TM fails with an OOME during checkpointing, its Tasks might not reach a proper failed state, because the failure handling mechanism is affected by the OOME, too. Fixed in 1.73 and 1.8.2. [12]

* [FLINK-13059] The CassandraSink might become deadlocked during shutdown, when an exception happens during asynchronously sending requests to Cassandra. A PR is available but needs a review. [13]

* [FLINK-12122] [1.6.4] [ 1.7.2] [ 1.8.1] Not a new bug, but a continuous source of questions: Since FLIP-6 tasks are not spread out evenly among all TaskManagers as before, which can result in very uneven loads if the number of available TaskSlots exceeds the parallelism of the submitted job. Right now it looks like a fix is targeted for 1.9.1 and 1.8.2 [14]

[11] https://issues.apache.org/jira/browse/FLINK-13063

[12] https://issues.apache.org/jira/browse/FLINK-12889

[13] https://issues.apache.org/jira/browse/FLINK-13059

[14] https://issues.apache.org/jira/browse/FLINK-12122

Events, Blog Posts, Misc

====================

* Flink Forward Europe early-bird ends on the 15th of July. [15]

* Upcoming Meetups

    * On 18th of July Christos Hadjinikolis is speaking at the "Big Data LDN Meetup" on "How real-time data processing is used for application in customer experience?" [16]

[15] https://berlin-2019.flink-forward.org/

[16] https://www.meetup.com/big-data-ldn/events/262638878/

Cheers,

Konstantin (@snntrable)

--