Prioritize DataStream

[Elias Levy]

I believe the answer to this question is "no", but I figure I might as well ask.  Is there a way to prioritize a stream?

The use case is prioritization of a control stream.  This is mostly needed on start-up, where a job might start consuming from the data stream before consuming from the control stream.

[Elias Levy]

Flink folks,

A response to the question below?

On Sat, Aug 19, 2017 at 11:02 AM, Elias Levy <[hidden email]> wrote:

I believe the answer to this question is "no", but I figure I might as well ask.  Is there a way to prioritize a stream?

The use case is prioritization of a control stream.  This is mostly needed on start-up, where a job might start consuming from the data stream before consuming from the control stream.

[Till Rohrmann]

Hi Elias,

sorry for the slow answer. You were right that the answer is currently no. 

However, people are currently working on changing the way the stream operators work. This will allow the operator to decide from which input to read next. Having such a functionality will enable us to implement proper side inputs which will also cover your use case.

At the moment, one thing you could try is to buffer events in operator state until you have fully consumed the other stream. Upon seeing all events from the other stream (needs a proper termination event you can detect) you can then replay all the buffered events. This could however entail quite a big user state.

Cheers,

Till

On Mon, Aug 21, 2017 at 8:14 PM, Elias Levy <[hidden email]> wrote:

Flink folks,

A response to the question below?

On Sat, Aug 19, 2017 at 11:02 AM, Elias Levy <[hidden email]> wrote:

I believe the answer to this question is "no", but I figure I might as well ask.  Is there a way to prioritize a stream?

The use case is prioritization of a control stream.  This is mostly needed on start-up, where a job might start consuming from the data stream before consuming from the control stream.