RawSchema as deserialization schema

[Swapnil Chougule]

I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read byte array as it is in datastream. I tried to use RawSchema as desrialization schema but couldn't find same 1.1.1.
I want to know whether I have to write my custom  implementation for same ?
Can somebody help me to sort out same ?

Also passing byte[] to next operator is supported as far as serialization is concerned ?

[Maximilian Michels]

Just implement DeserializationSchema and return the byte array from
Kafka. Byte array serialization poses no problem to the Flink
serialization.

On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
<[hidden email]> wrote:
> I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read
> byte array as it is in datastream. I tried to use RawSchema as
> desrialization schema but couldn't find same 1.1.1.
> I want to know whether I have to write my custom  implementation for same ?
> Can somebody help me to sort out same ?
>
> Also passing byte[] to next operator is supported as far as serialization is
> concerned ?

[Swapnil Chougule]

Thanks Maximilian. I implemented same & it worked for me. I was under impression that RawSchema is available from flink.

Regards,

Swapnil

On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <[hidden email]> wrote:

Just implement DeserializationSchema and return the byte array from
Kafka. Byte array serialization poses no problem to the Flink
serialization.

On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
<[hidden email]> wrote:
> I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read
> byte array as it is in datastream. I tried to use RawSchema as
> desrialization schema but couldn't find same 1.1.1.
> I want to know whether I have to write my custom  implementation for same ?
> Can somebody help me to sort out same ?
>
> Also passing byte[] to next operator is supported as far as serialization is
> concerned ?

[Swapnil Chougule]

It will be good to have RawSchema as one of the deserialization schema in streaming framework (like SimpleStringSchema).

Many use cases needs data in byte array format after reading from source like kafka.

Any inputs for same ?

On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <[hidden email]> wrote:

Thanks Maximilian. I implemented same & it worked for me. I was under impression that RawSchema is available from flink.

Regards,

Swapnil

On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <[hidden email]> wrote:

Just implement DeserializationSchema and return the byte array from
Kafka. Byte array serialization poses no problem to the Flink
serialization.

On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
<[hidden email]> wrote:
> I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read
> byte array as it is in datastream. I tried to use RawSchema as
> desrialization schema but couldn't find same 1.1.1.
> I want to know whether I have to write my custom  implementation for same ?
> Can somebody help me to sort out same ?
>
> Also passing byte[] to next operator is supported as far as serialization is
> concerned ?

[Stephan Ewen]

/cc Robert, he is looking into extending the Kafka Connectors to support more of Kafka's direct utilities

On Thu, Sep 22, 2016 at 3:17 PM, Swapnil Chougule <[hidden email]> wrote:

It will be good to have RawSchema as one of the deserialization schema in streaming framework (like SimpleStringSchema).

Many use cases needs data in byte array format after reading from source like kafka.

Any inputs for same ?

On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <[hidden email]> wrote:

Thanks Maximilian. I implemented same & it worked for me. I was under impression that RawSchema is available from flink.

Regards,

Swapnil

On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <[hidden email]> wrote:

Just implement DeserializationSchema and return the byte array from
Kafka. Byte array serialization poses no problem to the Flink
serialization.

On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
<[hidden email]> wrote:
> I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read
> byte array as it is in datastream. I tried to use RawSchema as
> desrialization schema but couldn't find same 1.1.1.
> I want to know whether I have to write my custom  implementation for same ?
> Can somebody help me to sort out same ?
>
> Also passing byte[] to next operator is supported as far as serialization is
> concerned ?

[Swapnil Chougule]

Hi Robert, May I know your inputs on same ?

Thanks,

Swapnil

On Thu, Sep 22, 2016 at 7:12 PM, Stephan Ewen <[hidden email]> wrote:

/cc Robert, he is looking into extending the Kafka Connectors to support more of Kafka's direct utilities

On Thu, Sep 22, 2016 at 3:17 PM, Swapnil Chougule <[hidden email]> wrote:

It will be good to have RawSchema as one of the deserialization schema in streaming framework (like SimpleStringSchema).

Many use cases needs data in byte array format after reading from source like kafka.

Any inputs for same ?

On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <[hidden email]> wrote:

Thanks Maximilian. I implemented same & it worked for me. I was under impression that RawSchema is available from flink.

Regards,

Swapnil

On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <[hidden email]> wrote:

Just implement DeserializationSchema and return the byte array from
Kafka. Byte array serialization poses no problem to the Flink
serialization.

On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
<[hidden email]> wrote:
> I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read
> byte array as it is in datastream. I tried to use RawSchema as
> desrialization schema but couldn't find same 1.1.1.
> I want to know whether I have to write my custom  implementation for same ?
> Can somebody help me to sort out same ?
>
> Also passing byte[] to next operator is supported as far as serialization is
> concerned ?

[rmetzger0]

The RawSchema was once part of Flink's Kafka connector. I've removed it because its implementation is trivial and I didn't expect that there are many people who need the schema (also, I think I saw people using a map() operator after the consumer to deserialize the byte[] into their formats).

As Stephan mentioned, I'm going to add some tooling so that people can directly use Kafka's Serialization classes, and they provide a "ByteArraySerializer" which is doing exactly the same as the "RawSchema"

On Mon, Sep 26, 2016 at 11:27 AM, Swapnil Chougule <[hidden email]> wrote:

Hi Robert, May I know your inputs on same ?

Thanks,

Swapnil

On Thu, Sep 22, 2016 at 7:12 PM, Stephan Ewen <[hidden email]> wrote:

/cc Robert, he is looking into extending the Kafka Connectors to support more of Kafka's direct utilities

On Thu, Sep 22, 2016 at 3:17 PM, Swapnil Chougule <[hidden email]> wrote:

It will be good to have RawSchema as one of the deserialization schema in streaming framework (like SimpleStringSchema).

Many use cases needs data in byte array format after reading from source like kafka.

Any inputs for same ?

On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <[hidden email]> wrote:

Thanks Maximilian. I implemented same & it worked for me. I was under impression that RawSchema is available from flink.

Regards,

Swapnil

On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <[hidden email]> wrote:

Just implement DeserializationSchema and return the byte array from
Kafka. Byte array serialization poses no problem to the Flink
serialization.

On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
<[hidden email]> wrote:
> I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read
> byte array as it is in datastream. I tried to use RawSchema as
> desrialization schema but couldn't find same 1.1.1.
> I want to know whether I have to write my custom  implementation for same ?
> Can somebody help me to sort out same ?
>
> Also passing byte[] to next operator is supported as far as serialization is
> concerned ?

[Swapnil Chougule]

Okay. Thanks for the update Robert.

On Mon, Sep 26, 2016 at 3:08 PM, Robert Metzger <[hidden email]> wrote:

The RawSchema was once part of Flink's Kafka connector. I've removed it because its implementation is trivial and I didn't expect that there are many people who need the schema (also, I think I saw people using a map() operator after the consumer to deserialize the byte[] into their formats).

As Stephan mentioned, I'm going to add some tooling so that people can directly use Kafka's Serialization classes, and they provide a "ByteArraySerializer" which is doing exactly the same as the "RawSchema"

On Mon, Sep 26, 2016 at 11:27 AM, Swapnil Chougule <[hidden email]> wrote:

Hi Robert, May I know your inputs on same ?

Thanks,

Swapnil

On Thu, Sep 22, 2016 at 7:12 PM, Stephan Ewen <[hidden email]> wrote:

/cc Robert, he is looking into extending the Kafka Connectors to support more of Kafka's direct utilities

On Thu, Sep 22, 2016 at 3:17 PM, Swapnil Chougule <[hidden email]> wrote:

It will be good to have RawSchema as one of the deserialization schema in streaming framework (like SimpleStringSchema).

Many use cases needs data in byte array format after reading from source like kafka.

Any inputs for same ?

On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <[hidden email]> wrote:

Thanks Maximilian. I implemented same & it worked for me. I was under impression that RawSchema is available from flink.

Regards,

Swapnil

On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <[hidden email]> wrote:

Just implement DeserializationSchema and return the byte array from
Kafka. Byte array serialization poses no problem to the Flink
serialization.

On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
<[hidden email]> wrote:
> I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read
> byte array as it is in datastream. I tried to use RawSchema as
> desrialization schema but couldn't find same 1.1.1.
> I want to know whether I have to write my custom  implementation for same ?
> Can somebody help me to sort out same ?
>
> Also passing byte[] to next operator is supported as far as serialization is
> concerned ?