Flink Statefun Byte Ingress

[Timothy Bess]

Hi,

So most of the examples of "module.yaml" files I've seen focus on protobuf ingress, but is there a way to just get bytes from Kafka? I want to integrate this with the rest of my codebase which uses JSON, but don't want to migrate to protobuf just yet. I'm not totally sure how it would work since function arguments seem to be encoded as an Any type which is a protobuf type string + some bytes, I guess the string would need to be some made up constant value and I'd just grab the bytes? Honestly just using bytes like is done with the state value might be a bit more flexible to work with.

Thanks,

Tim

[Igal Shilman]

Hi Tim,

You are correct, currently the argument to a remote function must be a Protobuf Any, however StateFun doesn't interpret the contents of that Any, and it would be passed as-is to the remote function.

As you mentioned in your email you can interpret the bytes as the bytes of a JSON string.

I hope it helps,

Igal.

On Wed, Sep 23, 2020 at 5:06 AM Timothy Bess <[hidden email]> wrote:

Hi,

So most of the examples of "module.yaml" files I've seen focus on protobuf ingress, but is there a way to just get bytes from Kafka? I want to integrate this with the rest of my codebase which uses JSON, but don't want to migrate to protobuf just yet. I'm not totally sure how it would work since function arguments seem to be encoded as an Any type which is a protobuf type string + some bytes, I guess the string would need to be some made up constant value and I'd just grab the bytes? Honestly just using bytes like is done with the state value might be a bit more flexible to work with.

Thanks,

Tim

[Timothy Bess]

Hi Igal,

Ah that definitely helps to know for Function -> Function invocations, but when doing Ingress via statefun how would that work? Is there a config I can set in the "module.yaml" to have it just pack arbitrary bytes into the Any?

Thanks,

Tim

On Wed, Sep 23, 2020 at 7:01 AM Igal Shilman <[hidden email]> wrote:

Hi Tim,

You are correct, currently the argument to a remote function must be a Protobuf Any, however StateFun doesn't interpret the contents of that Any, and it would be passed as-is to the remote function.

As you mentioned in your email you can interpret the bytes as the bytes of a JSON string.

I hope it helps,

Igal.

On Wed, Sep 23, 2020 at 5:06 AM Timothy Bess <[hidden email]> wrote:

Hi,

So most of the examples of "module.yaml" files I've seen focus on protobuf ingress, but is there a way to just get bytes from Kafka? I want to integrate this with the rest of my codebase which uses JSON, but don't want to migrate to protobuf just yet. I'm not totally sure how it would work since function arguments seem to be encoded as an Any type which is a protobuf type string + some bytes, I guess the string would need to be some made up constant value and I'd just grab the bytes? Honestly just using bytes like is done with the state value might be a bit more flexible to work with.

Thanks,

Tim

[Igal Shilman]

Hi,

For ingress, we don't look at the content at all, we put the bytes "as-is" into the Any's value field, and we set the typeUrl field

with whatever was specified in the module.yaml.

See here for example: https://github.com/apache/flink-statefun/blob/master/statefun-examples/statefun-python-k8s-example/module.yaml#L41

Hope that helps,

Igal.

On Wed, Sep 23, 2020 at 3:09 PM Timothy Bess <[hidden email]> wrote:

Hi Igal,

Ah that definitely helps to know for Function -> Function invocations, but when doing Ingress via statefun how would that work? Is there a config I can set in the "module.yaml" to have it just pack arbitrary bytes into the Any?

Thanks,

Tim

On Wed, Sep 23, 2020 at 7:01 AM Igal Shilman <[hidden email]> wrote:

Hi Tim,

You are correct, currently the argument to a remote function must be a Protobuf Any, however StateFun doesn't interpret the contents of that Any, and it would be passed as-is to the remote function.

As you mentioned in your email you can interpret the bytes as the bytes of a JSON string.

I hope it helps,

Igal.

On Wed, Sep 23, 2020 at 5:06 AM Timothy Bess <[hidden email]> wrote:

Hi,

So most of the examples of "module.yaml" files I've seen focus on protobuf ingress, but is there a way to just get bytes from Kafka? I want to integrate this with the rest of my codebase which uses JSON, but don't want to migrate to protobuf just yet. I'm not totally sure how it would work since function arguments seem to be encoded as an Any type which is a protobuf type string + some bytes, I guess the string would need to be some made up constant value and I'd just grab the bytes? Honestly just using bytes like is done with the state value might be a bit more flexible to work with.

Thanks,

Tim