[ANNOUNCE] Weekly Community Update 2020/09

[Konstantin Knauf-2]

Dear community,

happy to share this week's community update. It was a relatively quiet week on the dev@ mailing list (mostly votes on previously covered FLIPs), but there is always something to share. Additionally, I have decided to also feature flink-packages.org in this newsletter going forward. Depending on the level of activity, I will cover newly added packages or introduce one of the existing packages.

Flink Development

==============

* [sql] Dawid has started a discussion to enable Tabla API/SQL sources to read columns from different parts of source records. With this it would, for example, be possible to read partition, timestamp or offset from a Kafka source record. Similarly, it would be possible to specify override partitioning when writing to Kafka or Kinesis. [1]

* [sql, python] FLIP-58 introduced Python UDFs in SQL and Table API. FLIP-79 added a Function DDL in Flink SQL to register Java & Scala UDFs in pure SQL. Based on these two FLIPs, Wei Zhon published FLIP-106 to also support Python UDFs in the SQL Function DDL. [2]

* [development] Chesnay started a discussion on Eclipse support for Apache Flink (framework) development. If you are using Eclipse as an Apache Flink contributor, please get involved in the thread. [3]

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-107-Reading-table-columns-from-different-parts-of-source-records-tp38277.html

[2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-106-Support-Python-UDF-in-SQL-Function-DDL-tp38107.html

[3] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Remove-Eclipse-specific-plugin-configurations-tp38255.html

Notable Bugs
==========

[FLINK-16262] [1.10.0] The FlinkKafkaProducer can not be used in EXACTLY_ONCE mode when using the user code classloader. For application cluster (per-job clusters) you can work around this issue by using the system classloader (user jar in lib/ directory). Will be fixed in 1.10.1. [4]

[4] https://issues.apache.org/jira/browse/FLINK-16262

flink-packages.org

=============

DTStack, a Chinese cloud technology company, has recently published FlinkX [5] on flink-packages.org. The documentation is Chinese only, but it seems to be a configuration-based integration framework based on Apache Flink with an impressive set of connectors.

[5] https://flink-packages.org/packages/flinkx

Events, Blog Posts, Misc
===================

* This week I stumbled across this Azure tutorial to use Event Hubs with Apache Flink. [6]

* Gökce Sürenkök has written a blog post on setting up a highly available Flink cluster on Kubernetes based on Zookeeper for Flink Master failover and HDFS as checkpoint storage. [7]

* Upcoming Meetups

    * On March 5th, Stephan Ewen will talk about Apache Flink Stateful Function at the Utrecht Data Engineering Meetup. [8]

    * On March 12th, Prateep Kumar will host an online event comparing Kafka Streams and Apache Flink [9].

    * On April 22, Ververica will host the next Apache Flink meetup in Berlin. [10]

    * Cloudera is hosting a couple of "Future of Data" events on stream processing with Apache Flink in
        * Vienna (March 4th, full-day workshop) [11]
        * Zurich (March 10th, full-day workshop) [12]

        * New Jersey (May 5th, meetup) [13]

[6] https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-flink-tutorial

[7] https://medium.com/hepsiburadatech/high-available-flink-cluster-on-kubernetes-setup-73b2baf9200e

[8] https://www.meetup.com/Data-Engineering-NL/events/268424399/

[9] https://www.meetup.com/apache-flink-aws-kinesis-hyd-india/events/268930388/

[10] https://www.meetup.com/Apache-Flink-Meetup/events/269005339/

[11] https://www.meetup.com/futureofdata-vienna/events/268418974/

[12] https://www.meetup.com/futureofdata-zurich/events/268423809/

[13] https://www.meetup.com/futureofdata-princeton/events/268830725/

Cheers,

Konstantin (@snntrable)

--