Side output question

[M Singh]

Hi:

I am replacing SplitOperator in my flink application with a simple processor with side outputs.

My questions is that does the main stream from which we get the side outputs need to have any events (ie, produced using by the using collector.collect) ?  Or can we have all the output as side outputs ? Also are there any pros and cons of at least one main collected output vs all side outputs ?

Thanks

Mans

[Arvid Heise-3]

Hi Mans,

there should be no issue to only have side-outputs in your operator. There should also be no big drawbacks. I guess mostly some metrics will not be properly populated, but you can always populate them manually or add new ones.

Best,

Arvid

On Mon, Dec 2, 2019 at 8:40 PM M Singh <[hidden email]> wrote:

Hi:

I am replacing SplitOperator in my flink application with a simple processor with side outputs.

My questions is that does the main stream from which we get the side outputs need to have any events (ie, produced using by the using collector.collect) ?  Or can we have all the output as side outputs ? Also are there any pros and cons of at least one main collected output vs all side outputs ?

Thanks

Mans

[M Singh]

Thanks Arvid for your answer.

Can you please point me to any documentation/reference as to which metrics might be impacted ? Also, let me know of any other pitfall.

Once again, I appreciate your help.

On Tuesday, December 10, 2019, 03:23:01 AM EST, Arvid Heise <[hidden email]> wrote:

Hi Mans,

there should be no issue to only have side-outputs in your operator. There should also be no big drawbacks. I guess mostly some metrics will not be properly populated, but you can always populate them manually or add new ones.

Best,

Arvid

On Mon, Dec 2, 2019 at 8:40 PM M Singh <[hidden email]> wrote:

Hi:

I am replacing SplitOperator in my flink application with a simple processor with side outputs.

My questions is that does the main stream from which we get the side outputs need to have any events (ie, produced using by the using collector.collect) ?  Or can we have all the output as side outputs ? Also are there any pros and cons of at least one main collected output vs all side outputs ?

Thanks

Mans

[Arvid Heise-3]

There is no clear reference as it's not a use case that has occurred yet. I'd be careful with all metrics related to output. Shuffle service should be fine [1] as side-output also go over it.

I wouldn't be surprised if currentOutputWatermark is not updated though.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#default-shuffle-service

On Tue, Dec 10, 2019 at 1:47 PM M Singh <[hidden email]> wrote:

Thanks Arvid for your answer.

Can you please point me to any documentation/reference as to which metrics might be impacted ? Also, let me know of any other pitfall.

Once again, I appreciate your help.

On Tuesday, December 10, 2019, 03:23:01 AM EST, Arvid Heise <[hidden email]> wrote:

Hi Mans,

there should be no issue to only have side-outputs in your operator. There should also be no big drawbacks. I guess mostly some metrics will not be properly populated, but you can always populate them manually or add new ones.

Best,

Arvid

On Mon, Dec 2, 2019 at 8:40 PM M Singh <[hidden email]> wrote:

Hi:

I am replacing SplitOperator in my flink application with a simple processor with side outputs.

My questions is that does the main stream from which we get the side outputs need to have any events (ie, produced using by the using collector.collect) ?  Or can we have all the output as side outputs ? Also are there any pros and cons of at least one main collected output vs all side outputs ?

Thanks

Mans