Status of Kafka011JsonTableSink for 1.4.0 release?

[Georgios Kaklamanos]

Hello,

I'm new to Flink and I'd like to use it along with Kafka 0.11, to do
some processing on a stream of JSON data.

I've been looking the docs of the Table API (and the relevant GitHub
code), and I see that recently there was a Kafka011JsonTableSource
submitted, which will be available in the Flink 1.4 release.

Would there also be a Kafka011JsonTableSink in the near future (1.4
release), or I'd have to fall back to Kafka 0.10 for now?

Thank you for your time.

Best Regards,
George



--
------------------------------------------------------------------
Georgios Kaklamanos
Research Assistant, e-Science Group, GWDG
mailto: [hidden email]
Telefon: [phone]
------------------------------------------------------------------
GWDG - Gesellschaft für wissenschaftliche
Datenverarbeitung mbH Göttingen
Am Faßberg 11, 37077 Göttingen, Germany

WWW: www.gwdg.de    mailto: [hidden email]
Phone: [phone]
Fax:   [phone]
------------------------------------------------------------------
Geschäftsführer: Prof. Dr. Ramin Yahyapour
Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen
Handelsregister-Nr. B 598
------------------------------------------------------------------
Zertifiziert nach ISO 9001
------------------------------------------------------------------

[Fabian Hueske-2]

Hi George,

Flink 1.4 will not include a KafkaTableSink for Kafka 0.11 but a DataStream API SinkFunction (KafkaProducer).

As an alternative to usingthe Kafka010TableSink, you can also convert the result Table into a DataStream and use the KafkaProducer for Kafka 0.11 to emit the DataStream.

We will hopefully add Kafka 0.11 TableSink with Flink 1.5 which is scheduled for early next year.

Best, Fabian

2017-11-26 2:31 GMT+01:00 Georgios Kaklamanos <[hidden email]>:

Hello,

I'm new to Flink and I'd like to use it along with Kafka 0.11, to do
some processing on a stream of JSON data.

I've been looking the docs of the Table API (and the relevant GitHub
code), and I see that recently there was a Kafka011JsonTableSource
submitted, which will be available in the Flink 1.4 release.

Would there also be a Kafka011JsonTableSink in the near future (1.4
release), or I'd have to fall back to Kafka 0.10 for now?

Thank you for your time.

Best Regards,
George



--
------------------------------------------------------------------
Georgios Kaklamanos
Research Assistant, e-Science Group, GWDG
mailto: [hidden email]
Telefon: [phone]
------------------------------------------------------------------
GWDG - Gesellschaft für wissenschaftliche
Datenverarbeitung mbH Göttingen
Am Faßberg 11, 37077 Göttingen, Germany

WWW: www.gwdg.de    mailto: [hidden email]
Phone: <a href="tel:%2B49%20%280%29%20551%20201-1510" value="[phone]">[phone]
Fax:   <a href="tel:%2B49%20%280%29%20551%20201-2150" value="[phone]">[phone]
------------------------------------------------------------------
Geschäftsführer: Prof. Dr. Ramin Yahyapour
Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen
Handelsregister-Nr. B 598
------------------------------------------------------------------
Zertifiziert nach ISO 9001
------------------------------------------------------------------

[Georgios Kaklamanos]

Hi Fabian,

Thanks for the answer.

I had seen the Kafka Producer but, from a quick look, I didn't seem to
find something like a JSON Serialization Schema, which I need since the
next app in my pipeline, expects to read the data in JSON.

So hoping for a TableJSONSink, I didn't look more into it. I'll check it
now.

Best,
George




On 11/27/2017 10:53 AM, Fabian Hueske wrote:

> Hi George,
>
> Flink 1.4 will not include a KafkaTableSink for Kafka 0.11 but a
> DataStream API SinkFunction (KafkaProducer).
> As an alternative to usingthe Kafka010TableSink, you can also convert
> the result Table into a DataStream and use the KafkaProducer for Kafka
> 0.11 to emit the DataStream.
>
> We will hopefully add Kafka 0.11 TableSink with Flink 1.5 which is
> scheduled for early next year.
>
> Best, Fabian
>
> 2017-11-26 2:31 GMT+01:00 Georgios Kaklamanos
> <[hidden email] <mailto:[hidden email]>>:
>
>     Hello,
>
>     I'm new to Flink and I'd like to use it along with Kafka 0.11, to do
>     some processing on a stream of JSON data.
>
>     I've been looking the docs of the Table API (and the relevant GitHub
>     code), and I see that recently there was a Kafka011JsonTableSource
>     submitted, which will be available in the Flink 1.4 release.
>
>     Would there also be a Kafka011JsonTableSink in the near future (1.4
>     release), or I'd have to fall back to Kafka 0.10 for now?
>
>     Thank you for your time.
>
>     Best Regards,
>     George
>
>
>
>     --
>     ------------------------------------------------------------------
>     Georgios Kaklamanos
>     Research Assistant, e-Science Group, GWDG
>     mailto: [hidden email] <mailto:[hidden email]>
>     Telefon: [phone]
>     ------------------------------------------------------------------
>     GWDG - Gesellschaft für wissenschaftliche
>     Datenverarbeitung mbH Göttingen
>     Am Faßberg 11, 37077 Göttingen, Germany
>
>     WWW: www.gwdg.de <http://www.gwdg.de>    mailto: [hidden email]
>     <mailto:[hidden email]>
>     Phone: [phone] <tel:%2B49%20%280%29%20551%20201-1510>
>     Fax:   [phone] <tel:%2B49%20%280%29%20551%20201-2150>
>     ------------------------------------------------------------------
>     Geschäftsführer: Prof. Dr. Ramin Yahyapour
>     Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger
>     Sitz der Gesellschaft: Göttingen
>     Registergericht: Göttingen
>     Handelsregister-Nr. B 598
>     ------------------------------------------------------------------
>     Zertifiziert nach ISO 9001
>     ------------------------------------------------------------------
>
>
>

--
------------------------------------------------------------------
Georgios Kaklamanos
Research Assistant, e-Science Group, GWDG
mailto: [hidden email]
Telefon: [phone]
------------------------------------------------------------------
GWDG - Gesellschaft für wissenschaftliche
Datenverarbeitung mbH Göttingen
Am Faßberg 11, 37077 Göttingen, Germany

WWW: www.gwdg.de    mailto: [hidden email]
Phone: [phone]
Fax:   [phone]
------------------------------------------------------------------
Geschäftsführer: Prof. Dr. Ramin Yahyapour
Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen
Handelsregister-Nr. B 598
------------------------------------------------------------------
Zertifiziert nach ISO 9001
------------------------------------------------------------------