Table/SQL Kafka Sink Question

[Pavel Ciorba]

Hi everyone!

Can I specify a message key using the Kafka sink in the Table/SQL API ?

The goal is to sink each row as JSON along side with a message key into Kafka.

I was achieving it using the Stream API by specifying a KeyedSerializationSchema using the serializeKey() method.

Thanks in advance!

[Chesnay Schepler]

Hello,

as far as i can this is not possible. I'm including Timo, maybe he can explain why this isn't supported.

On 26.03.2018 21:56, Pavel Ciorba wrote:

Hi everyone!

Can I specify a message key using the Kafka sink in the Table/SQL API ?

The goal is to sink each row as JSON along side with a message key into Kafka.

I was achieving it using the Stream API by specifying a KeyedSerializationSchema using the serializeKey() method.

Thanks in advance!

[Alexx]

That's what I concluded as well after checking the docs and source code.

I'm thinking to add another job using the Stream API (where it is possible), that will ingest the data resulted from by Table/SQL API job, and that will add the message key into Kafka.

On 27 March 2018 at 12:55, Chesnay Schepler <[hidden email]> wrote:

Hello,

as far as i can this is not possible. I'm including Timo, maybe he can explain why this isn't supported.

On 26.03.2018 21:56, Pavel Ciorba wrote:

Hi everyone!

Can I specify a message key using the Kafka sink in the Table/SQL API ?

The goal is to sink each row as JSON along side with a message key into Kafka.

I was achieving it using the Stream API by specifying a KeyedSerializationSchema using the serializeKey() method.

Thanks in advance!

[Timo Walther]

Hi Alexandru,

the KafkaTableSink does not expose all features of the underlying DataStream API. Either you convert your table program to the DataStream API for the sink operation or you just extend a class like Kafka010JsonTableSink and customize it.

Regards,
Timo


Am 27.03.18 um 11:59 schrieb Alexandru Gutan:

That's what I concluded as well after checking the docs and source code.

I'm thinking to add another job using the Stream API (where it is possible), that will ingest the data resulted from by Table/SQL API job, and that will add the message key into Kafka.

On 27 March 2018 at 12:55, Chesnay Schepler <[hidden email]> wrote:

Hello,

as far as i can this is not possible. I'm including Timo, maybe he can explain why this isn't supported.

On 26.03.2018 21:56, Pavel Ciorba wrote:

Hi everyone!

Can I specify a message key using the Kafka sink in the Table/SQL API ?

The goal is to sink each row as JSON along side with a message key into Kafka.

I was achieving it using the Stream API by specifying a KeyedSerializationSchema using the serializeKey() method.

Thanks in advance!