Multiple side outputs of same type?

[Alex Cruise]

Hey folks,

I have a program that demultiplexes input records from a shared prefix stream onto some number of suffix streams, which are allocated on boot based on configuration.

At the moment I'm just duplicating the input records, and filtering out the wrong records in each suffix stream, but it's not super efficient... Unfortunately, from what I can see, using side outputs isn't an option because each output tag has a single type parameter, and the output record is dispatched based on its runtime type.

Is there a better way to do this?

Thanks!

-0xe1a

[Arvid Heise-3]

Hi Alex,

not entirely sure how you reached your conclusion but afaik side output is dispatched through the output tag.

There are even tests in the code base [1] that use multiple outputs of the same type.

[1] https://github.com/apache/flink/blob/1a08548a209167cafeeba1ce602fe8d542994be5/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/StreamOperatorChainingTest.java#L179

On Fri, Dec 18, 2020 at 7:52 PM Alex Cruise <[hidden email]> wrote:

Hey folks,

I have a program that demultiplexes input records from a shared prefix stream onto some number of suffix streams, which are allocated on boot based on configuration.

At the moment I'm just duplicating the input records, and filtering out the wrong records in each suffix stream, but it's not super efficient... Unfortunately, from what I can see, using side outputs isn't an option because each output tag has a single type parameter, and the output record is dispatched based on its runtime type.

Is there a better way to do this?

Thanks!

-0xe1a

--

[Alex Cruise]

uh... OK, thanks! 😅

-0xe1a

On Fri, Dec 18, 2020 at 11:20 AM Arvid Heise <[hidden email]> wrote:

Hi Alex,

not entirely sure how you reached your conclusion but afaik side output is dispatched through the output tag.

There are even tests in the code base [1] that use multiple outputs of the same type.

[1] https://github.com/apache/flink/blob/1a08548a209167cafeeba1ce602fe8d542994be5/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/StreamOperatorChainingTest.java#L179

On Fri, Dec 18, 2020 at 7:52 PM Alex Cruise <[hidden email]> wrote:

Hey folks,

I have a program that demultiplexes input records from a shared prefix stream onto some number of suffix streams, which are allocated on boot based on configuration.

At the moment I'm just duplicating the input records, and filtering out the wrong records in each suffix stream, but it's not super efficient... Unfortunately, from what I can see, using side outputs isn't an option because each output tag has a single type parameter, and the output record is dispatched based on its runtime type.

Is there a better way to do this?

Thanks!

-0xe1a

--

Arvid Heise | Senior Java Developer

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng