[ANNOUNCE] Weekly Community Update 2020/31-34
Locked
[ANNOUNCE] Weekly Community Update 2020/31-34
 
|
Dear community, The "weekly" community update is back after a short summer break! This time I've tried to cover most of what happened during the last four weeks, but I might pick up some older topics in the next weeks' updates, too. Activity on the dev@ mailing list has picked up quite a bit as feature development & design for the next releases of Apache Flink and Apache Flink Stateful Functions is going at full steam. In detail: Flink Development ============== * [releases] [Flink 1.12] The work on Flink 1.12 is well underway with feature freeze planned for end of October [1]. Our release managers Robert & Dian are periodically reminding the developer community of current blockers to reduce time during release testing for this release [2]. * [releases] [Stateful Functions 2.2] Igal has started a discussion releasing Stateful Functions 2.2. soon (proposed feature freeze: September 10). The most notable feature is maybe the option to embed a stateful functions module in a DataStream program via DataStream Ingress/Egress. Checkout [3] for a full list of the planned features. * [releases] [Flink 1.10] Flink 1.10.2 was released. [4] * [apis] Besides the Stateful Functions API, Flink currently has three top-level APIs: DataStream (streaming), DataSet (batch) and TableAPI/SQL (unified). A major step towards the goal of a truly unified batch and stream processing engine is the unification of the DataStream/DataSet APIs. This is one of the main topics of the upcoming release(s), specifically: * Aljoscha has published FLIP-131 [5] proposing to deprecate and eventually drop the DataSet API. In order to still support the same breadth of use cases, we need to make sure that all its use cases are covered by the two remaining APIs: a unified DataStream API and the Table API. These changes are not part of FLIP-131 itself, but are covered in other FLIPs, which already exist (like FLIP-27 [6] or FLIP-129 [7]) or will be published over the next few weeks like FLIP-134 (see below). [8] * Most importantly, FLIP-134 [9] discusses how the DataStream API could be used to efficiently execute batch workloads in the future. In essence the FLIP proposes to introduce a BATCH and a STREAMING execution mode for DataStream programs. The STREAMING mode corresponds to the current behavior, while the BATCH mode adjusts the behavior in various areas to fit the requirements of batch processing, e.g. pipelined scheduling with region failover, blocking shuffles, no checkpointing, no watermarks, ... [10] * [apis] Time proposes FLIP-136 to improve the interoperability between the Data Stream and Table API. The FLIP covers the conversion between DataStream <-> Table (incl. cnangelong streams, watermarks, etc.) as well as more additional support for working with the Row type in the DataStream API. [11] * [datastream api] Dawid proposes to remove a set of deprecated methods from the DataStream API. [12] * [runtime] Yuan Mei has started a discussion on FLIP-135 to introduce task-local recovery. The FLIP is about the introduction of a new failover/recovery strategy for Flink Jobs, that trades consistency for availability. Specifically, in the case of approximate task-local recovery the failure of some tasks would not trigger a restart of the rest of the job, but in turn you can expect data loss or duplication. [13] * [python] Xingbo Huang proposes to extend the support of Pandas/vectorized functions from scalar functions to aggregate functions. For more details on Pandas support on PyFlink see the blog post linked below. [14] * [connectors] Aljoscha has started a discussion on dropping support for Kafka 0.10/0.11 in Flink 1.12+. [15] * [connectors] Robert has revived the discussion on adding support for Hbase 2.3.x. There is a consensus to add the HBase 2.x connector Apache Flink, but no consensus yet on whether to move the existing HBase 1.x from the Flink project to Apache Bahir, too. [16] ============== Jark has recently published a set of Flink connectors (DataStream & Table API/SQL) that allow to ingest the changelog of MySQL and Postgres without additional tools like Kafka or Debezium. [17] Notable Bugs ========== To be honest, I did not search through every bug ticket created over the last four weeks, only the last seven days, and I did not find anything particularly notable. So, I'll leave you without any bug reports this time. Events, Blog Posts, Misc =================== * David Anderson is now an Apache Flink committer. Congrats! [18] * There have been a couple blog posts on the Flink blog recently that highlight some of the features added in latest release: * Monitoring and Controlling Networks of IoT Devices with Flink Stateful Functions [21] * The schedule for Flink Forward Global is live [23]. The event is free and you can already register under [24]. Cheers, Konstantin |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |