[ANNOUNCE] Weekly Community Update 2020/44-45
Locked
[ANNOUNCE] Weekly Community Update 2020/44-45
 
|
Dear community, two weeks have passed again and I am happy two share another update with news on Flink 1.12, Flink 1.11.3 and the release of Stateful Functions 2.2.1. As everyone has been finishing the last bit and pieces of Flink 1.12, there are only a handful of new initiatives to cover this time including a so-called hybrid source and incremental checkpointing for the heap-based statebackends. Flink Development ============== * [releases] The feature freeze for Flink 1.12 happened on Monday and a first non-voting/testing release candidate has been published. [1] The community is collecting (manual) testing tasks in the wiki [2]. * [releases] There are still a few blockers to resolve before a first release candidate for Flink 1.11.3 is published. [3] * [releases] Stateful Functions 2.2.0 experiences a critical bug that causes restore from checkpoints or savepoints to fail in certain situations (FLINK-19692). The proper fix will be included in Flink 1.11.3. Since Flink 1.11.3 still takes a few days, Gordon proposed to release Stateful Functions 2.2.1 right away, that already fixes the issues when the framework version across snapshot creation and restore is the same. The release has already been approved and will be announced shortly. [4,5] * [sql] Jark has updated FLIP-145 after a round of offline discussions. The new windowing syntax will now also support session windows, propagate the window time as a time attribute and the FLIP proposes to deprecate the current GROUP BY window aggregation syntax. A new vote has been started based on the recent changes to the FLIP. [6,7] * [connectors] Nicholas Jiang has published a FLIP to support "Hybrid Sources". A Hybrid Source consists of multiple regular sources that are read from one after the other. Hybrid sources aim to make reprocessing/backfilling of data easier if the data is already distributed over multiple systems (e.g. last 14 days in Kafka, history in S3). [8] * [statebackends] Roman has published FLIP-151 to support incremental snapshotting for the heap-based state backend. Currently, incremental snapshotting is only supported by the RocksDBStatebackend. The HeapStatebackend is still preferable in a few situations and support for incremental checkpointing would overcome its largest limitation (besides limiting the state size to memory). [9] * [docker] In contrast to what I wrote would become the outcome of the discussion to make jemalloc the default memory allocator in the Apache Flink docker image, jemalloc will indeed become the default. [10] Notable Bugs ========== * [FLINK-19970][1.11.2] There might be a state leak in the CEP library that leads to an ever growing state size. I don't think this has been reproduced yet, but for anyone using the CEP library this is an interesting one to watch. [10] * [FLINK-20033] [1.11.2] [1.10.2] When a Job Master is stopped (which happens if the Dispatcher loses leadership) the current execution of its Job is failed, which can lead to data loss if the number of restarts are depleted. Fixed for 1.11.3 & 1.10.3. [11] Events, Blog Posts, Misc =================== * Congxian Qiu is now an Apache Flink Committer. Congratulations! [12] * Xianghu Wang has published a blog post outlining Apache Hudi's transition away from a Spark-only and towards a Flink-first architecture. [13] * Fred Teunissen & Erik de Nooij describe their solution to deal with event-time skew when ingesting data from heterogeneous Kafka partitions within one Flink Job on the Ververica Blog. [14] Cheers, Konstantin |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |