Side Output from AsyncFunction

[Mikhail Pryakhin-2]

Hello Flink experts!

My streaming pipeline makes async IO calls via the recommended AsyncFunction. The pipeline evolves and I've encountered a requirement to side output additional events from the function.

As it turned out the side output feature is only available in the following functions:

• ProcessFunction 
• CoProcessFunction
• ProcessWindowFunction
• ProcessAllWindowFunction

Just curious whether there is any known approach of utilizing "side output" feature from the AsyncFunction?

From my perspective, the only feasible way of solving this task is to output a processing event and an optional "side output" as a tuple from the AsyncFunction and then process the resulting stream with a ProcessFunction to handle the required logic. But this approach seems to be overcomplicated. 

Am I missing something? Any help/ideas are much appreciated!

Cheers,

Mike Pryakhin

[Ken Krugler]

Hi Mike,

1. Depending on what you need the side output for, you can use metrics to track some things.

But yes, that’s a very limited subset of all use cases.

2. As you mentioned, you could output a combo record.

Using an Either<regular output type, side output type> is a common approach.

But you don’t need a process function, you can use the .split() operator to create a SplitStream, and then .select() either the regular or the “side output” stream.

— Ken

On Mar 11, 2019, at 9:40 AM, Mikhail Pryakhin <[hidden email]> wrote:

Hello Flink experts!

My streaming pipeline makes async IO calls via the recommended AsyncFunction. The pipeline evolves and I've encountered a requirement to side output additional events from the function.

As it turned out the side output feature is only available in the following functions:

• ProcessFunction 
• CoProcessFunction
• ProcessWindowFunction
• ProcessAllWindowFunction

Just curious whether there is any known approach of utilizing "side output" feature from the AsyncFunction?

From my perspective, the only feasible way of solving this task is to output a processing event and an optional "side output" as a tuple from the AsyncFunction and then process the resulting stream with a ProcessFunction to handle the required logic. But this approach seems to be overcomplicated. 

Am I missing something? Any help/ideas are much appreciated!

Cheers,

Mike Pryakhin

--------------------------

Ken Krugler
[phone]
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra