happy to finally share another community update with you. With Flink Forward and the release of Flink SQL on Ververica Platform I was a bit too busy the previous weeks. I'll try to return to the weekly cadence going forward again. This time we have a lot of ground to cover. With the feature freeze for Flink 1.12 around the corner, this includes some last FLIPs for this release as well as the first FLIPs that will go into Flink 1.13.
Flink Development
==============
* [releases] It looks like the feature freeze for Flink 1.12 will be set to Monday, 2nd November. [1]
* [releases] Stateful Function 2.2.0 was released. [2] Please check out the release blog post [3] for an overview of the new features.
* [releases] Gordon started a discussion on the release of Flink 1.11.3. Gordon and Xintong will be release managers. Looks like this will happen soonish. [4]
* [releases] flink-shaded 12.0 was released. [5]
* [sql] Jark has started the discussion on FLIP-145 to support windowing table-valued functions. This FLIP introduces a new way to do windowing in Flink SQL, which would eventually supersede the current grouping windows. It adds four new windowing table-valued functions for TUMBLE, HOP, CUMULATE and SESSION windows. These functions return a new table with the same schema as the original table + a "window_start" and "window_end" column. For every window that a row falls into, one row is added to the output table. This windowing API provides a better integration with other parts of Flink SQL and is hopefully easier to pick up for SQL users than the current syntax. [6]
* [sql] Godfrey has shared a design document to use the multiple-input stream operator added in FLIP-92 to replace forward shuffles in Flink SQL by local function calls. [7]
* [sql] [connectors] Jingsong started a discussion and subsequent vote on FLIP-146. FLIP-146 extends the new TableSource/TableSink interfaces so that all existing table sources and sinks can be migrated to the new interfaces. Specifically, it adds the option to create a Table Source/Sink from a DataStream. [8]
* [sql] [connectors] Shengkai started the discussion and subsequently the vote on FLIP-149 to support reading/writing Kafka topics in upsert mode with Flink SQL (aka "Support for Compacted Kafka Topics"). With this FLIP users will be able to write changelog streams to (presumably compacted) Kafka topics and Kafka topics can be interpreted as changelog streams (regular record=upsert, record with null value=delete). [9]
* [deployment] Robert started a discussion on dropping Mesos support in one of the upcoming releases (earliest Flink 1.13). A lot of support, but also concerns by current users. So, please if you are Flink on Mesos, let the community know, so that we can make an informed decision. [10]
* [connectors] Dominik Wosinksi has started a discussion on a MongoDB (streaming) connector for Apache Flink. There have been tickets/PRs in Flink and Bahir before and he would like to come to some kind of conclusion. [11]
* [connectors] Darion Yaphet has asked if it were possible to provide a connector for the Open Source distributed graph database Nebula Graph. [12]
* [connectors] Kostas proposes to remove the long time deprecated flink-connector-filesystem. No conclusion yet. [13]
* [runtime] Yingjie Cao proposed to add a sort-merge-based blocking shuffle to Apache Flink in addition to the hash-style blocking shuffle implementation that Flink currently uses. The vote is currently ongoing. [14]
* [docker] Yun Tang started a discussion on using jemalloc as memory allocator for debian based Flink Docker images. The conclusion seems to be to make this configurable via an environment variable and to keep glibc malloc as the default. [15]
* [development process] Aljoscha proposes to use an automatic code-formatted like spotless going forward. Generally, a lot of support to use a code-formatted and no strong opinions on the actual code style to be used. There is some discussion on whether to do this incrementally or all at once. [16]
Notable Bugs
==========
I've limited my research to the last seven days.
* [FLINK-19790] [1.11.2] Writing MAP column with flink-json produces incorrect results. Fixed for 1.11.3. [17]
* [FLINK-19711] [1.11.2] [1.10.2] "csv" format does not ignore all parsing errors even when told so. Fixed for 1.11.3 and 1.10.3. [18]
* [FLINK-19557] [1.11.2] [1.10.2] When a following job manager loses its connection to Zookeeper and subsequently reconnects, it might miss the fact that the leader has not changed in the meantime is stuck in "leader election". Please check the ticket for a more precise description. Fixed for 1.10.3 and 1.11.3. [19]
Events, Blog Posts, Misc
===================
* Zhu Zhu is now a member of the Apache Flink PMC. [20]
* There are two newish blog post on the Flink blog:
* Arvid and Stephan published a first part of a blog post series on the motivation and development of unaligned checkpoints. [21]
* Gordon published a blog post that looks under the hood of Apache Flink stateful functions, e.g. explaining the communication protocol between remote functions and the stateful functions runtime. [22]
* Flink Forward Global took place last week. The recordings of most of the talks will be available soon.
Cheers,
Konstantin
--
Locked
