Hi,
I'm not quite sure if I understand your problem correctly. But your use
case sounds like a typical application of a union operation.
What do you mean with "knowledge of their destination sink"? The
operators don't need to be aware of the destination sink. The only thing
that needs to be coordinated is the result data type of each operation.
So either you force each operation to have a unified type or you create
a unified type before declaring the sink.
Or is every operator an independent Flink job? Maybe you can show us a
skeleton of your pipeline?
Regards,
Timo
Am 08.11.18 um 01:20 schrieb burgesschen:
> Hi Guys! I'm designing a topology where multiple operators should forward the
> messages to the same sink.
>
>
> For example I have Operator A,B,C,D,E. I want A,B,C to forward to Sink1 and
> D, E to forward to Sink2.
>
> My options are
>
> 1. Union A, B and C. then add Sink1 to them. Similarly for D and E. However,
> the current framework out team has builds each operator individually. There
> is nothing outside of the operators
> that has the knowledge of their destination sink. It means we need to build
> something on the job level to union the operators.
>
> 2. have each operator output to a side output tag. A,B, and C will output to
> tag "sink1", And have a singleton sink1 to consume from tag "sink1".
> Similarly for sink2. My concern here is that 'it feels hacky', since those
> messages are not really side outputs.
>
> is this a legitimate use case for output tag or not? Is there a better way
> to achieve this? Thank you!
>
>
>
>
>
>
> --
> Sent from:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Locked
