Storm topologies compatibility and exactly-once

[Olivier Mallassi]

hello all

I was reading the apache flink documentation and was particularly interested in two things

- compatibility with storm api. AFAIU(i need to do more tests), you can reuse storm defined topologies and run them on a Flink cluster. I was wondering if this is "still a bêta" or production ready? I have not looked at the implementation but I also assume there is a kind of bridge between the Storm APIs and the Flink internals so that you only exexute the spout/bolt code and do not inherit the Storm internals, right?

- exactly once semantic. I have to say this is a great feature :). I was wondering if this semantic is still available when running a Storm defined topology in a Flink cluster (cf my previous point) 

Thanks a lot for your help

Cheers

Olivier

[Maximilian Michels]

Hi Olivier,

Regarding the general question, please have a look at the
documentation: https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/storm_compatibility.html
Yes, you may reuse your existing spouts/bolts, the Storm runtime is
exchanged for the Flink runtime.

Exactly-once: Within Flink this works as expected but the spouts/bolts
are not checkpointed at the moment. That means that state won't be
restored after a failure of the job. Thus, you end up with only
at-least-once. We plan to change this very soon.

Let us know how your experiments go.

Cheers,
Max



On Sun, Mar 27, 2016 at 11:31 AM, Olivier Mallassi
<[hidden email]> wrote:

> hello all
>
> I was reading the apache flink documentation and was particularly interested
> in two things
> - compatibility with storm api. AFAIU(i need to do more tests), you can
> reuse storm defined topologies and run them on a Flink cluster. I was
> wondering if this is "still a bêta" or production ready? I have not looked
> at the implementation but I also assume there is a kind of bridge between
> the Storm APIs and the Flink internals so that you only exexute the
> spout/bolt code and do not inherit the Storm internals, right?
> - exactly once semantic. I have to say this is a great feature :). I was
> wondering if this semantic is still available when running a Storm defined
> topology in a Flink cluster (cf my previous point)
>
> Thanks a lot for your help
>
> Cheers
>
> Olivier

[Olivier Mallassi]

Thanks for the precision Maximilian. 

On Tue, Mar 29, 2016 at 11:27 AM, Maximilian Michels <[hidden email]> wrote:

Hi Olivier,

Regarding the general question, please have a look at the
documentation: https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/storm_compatibility.html
Yes, you may reuse your existing spouts/bolts, the Storm runtime is
exchanged for the Flink runtime.

Exactly-once: Within Flink this works as expected but the spouts/bolts
are not checkpointed at the moment. That means that state won't be
restored after a failure of the job. Thus, you end up with only
at-least-once. We plan to change this very soon.

Let us know how your experiments go.

Cheers,
Max

On Sun, Mar 27, 2016 at 11:31 AM, Olivier Mallassi
<[hidden email]> wrote:
> hello all
>
> I was reading the apache flink documentation and was particularly interested
> in two things
> - compatibility with storm api. AFAIU(i need to do more tests), you can
> reuse storm defined topologies and run them on a Flink cluster. I was
> wondering if this is "still a bêta" or production ready? I have not looked
> at the implementation but I also assume there is a kind of bridge between
> the Storm APIs and the Flink internals so that you only exexute the
> spout/bolt code and do not inherit the Storm internals, right?
> - exactly once semantic. I have to say this is a great feature :). I was
> wondering if this semantic is still available when running a Storm defined
> topology in a Flink cluster (cf my previous point)
>
> Thanks a lot for your help
>
> Cheers
>
> Olivier