Side output from Flink Sink

[Victor Villa Dev]

I'd like know if there's a way to generate a side output and/or sink to an alternate kafka topic from within a Sink?

The use case is the datastream sinks to a storage and on particular failed attempts I'd like to deadletter to a kafka topic.

Any suggestions?

Thanks

[vino yang]

Hi Victor,

Firstly, you can get your side output stream via OutputTag. Please refer to the official documentation[1].

Then, specify a sink for your side output stream. Of course, you can specify a Kafka sink.

Best,

Vino

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html

Victor Villa Dev <[hidden email]> 于2019年11月25日周一 上午2:27写道：

I'd like know if there's a way to generate a side output and/or sink to an alternate kafka topic from within a Sink?

The use case is the datastream sinks to a storage and on particular failed attempts I'd like to deadletter to a kafka topic.

Any suggestions?

Thanks

[Victor Villa Dev]

Hi Vino,

Thanks a lot for your reply!
However I'm not quite sure my question was clear enough.
I'm aware I can create/get side outputs using output tags from within operators (Process Functions) as documentation also states.

The main point in my question is wether creating a sideo output is even possible from within a Sink?
if so, would you mind pointing to an examples on how to correctly get the context necessary to add the "output" from within the "invoke()" method.
In case it isn't what are the usual/suggested strategies?

I know the Sink is usually the "last" portion of a data stream as its name indicates, but I was wondering if for some reason something can't be sinked (after retries, etc), what is the usual way to deal with such cases?

Thanks again for your kind support.

On 2019/11/25 02:23:15, vino yang <[hidden email]> wrote:

> Hi Victor,
>
> Firstly, you can get your side output stream via OutputTag. Please refer to
> the official documentation[1].
> Then, specify a sink for your side output stream. Of course, you can
> specify a Kafka sink.
>
> Best,
> Vino
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
>
> Victor Villa Dev <[hidden email]> 于2019年11月25日周一 上午2:27写道：
>
> > I'd like know if there's a way to generate a side output and/or sink to an
> > alternate kafka topic from within a Sink?
> > The use case is the datastream sinks to a storage and on particular failed
> > attempts I'd like to deadletter to a kafka topic.
> > Any suggestions?
> >
> > Thanks
> >
>

[vino yang]

Hi Victor,

Currently, it seems the "side output" feature does not been supported by the streaming sink.

IMO, you can customize your sink via selecting different types of events to output to different places.

WDYT?

Best,

Vino

Victor Villa Dev <[hidden email]> 于2019年11月25日周一 下午1:37写道：

Hi Vino,

Thanks a lot for your reply!
However I'm not quite sure my question was clear enough.
I'm aware I can create/get side outputs using output tags from within operators (Process Functions) as documentation also states.

The main point in my question is wether creating a sideo output is even possible from within a Sink?
if so, would you mind pointing to an examples on how to correctly get the context necessary to add the "output" from within the "invoke()" method.
In case it isn't what are the usual/suggested strategies?

I know the Sink is usually the "last" portion of a data stream as its name indicates, but I was wondering if for some reason something can't be sinked (after retries, etc), what is the usual way to deal with such cases?

Thanks again for your kind support.

On 2019/11/25 02:23:15, vino yang <[hidden email]> wrote:
> Hi Victor,
>
> Firstly, you can get your side output stream via OutputTag. Please refer to
> the official documentation[1].
> Then, specify a sink for your side output stream. Of course, you can
> specify a Kafka sink.
>
> Best,
> Vino
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
>
> Victor Villa Dev <[hidden email]> 于2019年11月25日周一 上午2:27写道：
>
> > I'd like know if there's a way to generate a side output and/or sink to an
> > alternate kafka topic from within a Sink?
> > The use case is the datastream sinks to a storage and on particular failed
> > attempts I'd like to deadletter to a kafka topic.
> > Any suggestions?
> >
> > Thanks
> >
>

[Arvid Heise-3]

Hi Victor,

you could implement your own SinkFunction that wraps the KafkaProducer. However, since you may need to check if the write operation is successful, you probably need to subclass KafkaProducer and implement your own error handling.

Best,

Arvid

On Mon, Nov 25, 2019 at 7:51 AM vino yang <[hidden email]> wrote:

Hi Victor,

Currently, it seems the "side output" feature does not been supported by the streaming sink.

IMO, you can customize your sink via selecting different types of events to output to different places.

WDYT?

Best,

Vino

Victor Villa Dev <[hidden email]> 于2019年11月25日周一 下午1:37写道：

Hi Vino,

Thanks a lot for your reply!
However I'm not quite sure my question was clear enough.
I'm aware I can create/get side outputs using output tags from within operators (Process Functions) as documentation also states.

The main point in my question is wether creating a sideo output is even possible from within a Sink?
if so, would you mind pointing to an examples on how to correctly get the context necessary to add the "output" from within the "invoke()" method.
In case it isn't what are the usual/suggested strategies?

I know the Sink is usually the "last" portion of a data stream as its name indicates, but I was wondering if for some reason something can't be sinked (after retries, etc), what is the usual way to deal with such cases?

Thanks again for your kind support.

On 2019/11/25 02:23:15, vino yang <[hidden email]> wrote:
> Hi Victor,
>
> Firstly, you can get your side output stream via OutputTag. Please refer to
> the official documentation[1].
> Then, specify a sink for your side output stream. Of course, you can
> specify a Kafka sink.
>
> Best,
> Vino
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
>
> Victor Villa Dev <[hidden email]> 于2019年11月25日周一 上午2:27写道：
>
> > I'd like know if there's a way to generate a side output and/or sink to an
> > alternate kafka topic from within a Sink?
> > The use case is the datastream sinks to a storage and on particular failed
> > attempts I'd like to deadletter to a kafka topic.
> > Any suggestions?
> >
> > Thanks
> >
>

[rmetzger0]

What do you mean by "from within a sink"? Do you have a custom sink?

If you want to write to different Kafka topics from the same sink, you can do that using a custom KafkaSerializationSchema. It allows you to return a ProducerRecord with a custom target topic set. (A Kafka sink can write to multiple topics).

Why can't you split the stream into "good" and "bad" records before the sink, and then define two different sinks?

On Thu, Nov 28, 2019 at 1:41 PM Arvid Heise <[hidden email]> wrote:

Hi Victor,

you could implement your own SinkFunction that wraps the KafkaProducer. However, since you may need to check if the write operation is successful, you probably need to subclass KafkaProducer and implement your own error handling.

Best,

Arvid

On Mon, Nov 25, 2019 at 7:51 AM vino yang <[hidden email]> wrote:

Hi Victor,

Currently, it seems the "side output" feature does not been supported by the streaming sink.

IMO, you can customize your sink via selecting different types of events to output to different places.

WDYT?

Best,

Vino

Victor Villa Dev <[hidden email]> 于2019年11月25日周一 下午1:37写道：

Hi Vino,

Thanks a lot for your reply!
However I'm not quite sure my question was clear enough.
I'm aware I can create/get side outputs using output tags from within operators (Process Functions) as documentation also states.

The main point in my question is wether creating a sideo output is even possible from within a Sink?
if so, would you mind pointing to an examples on how to correctly get the context necessary to add the "output" from within the "invoke()" method.
In case it isn't what are the usual/suggested strategies?

I know the Sink is usually the "last" portion of a data stream as its name indicates, but I was wondering if for some reason something can't be sinked (after retries, etc), what is the usual way to deal with such cases?

Thanks again for your kind support.

On 2019/11/25 02:23:15, vino yang <[hidden email]> wrote:
> Hi Victor,
>
> Firstly, you can get your side output stream via OutputTag. Please refer to
> the official documentation[1].
> Then, specify a sink for your side output stream. Of course, you can
> specify a Kafka sink.
>
> Best,
> Vino
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
>
> Victor Villa Dev <[hidden email]> 于2019年11月25日周一 上午2:27写道：
>
> > I'd like know if there's a way to generate a side output and/or sink to an
> > alternate kafka topic from within a Sink?
> > The use case is the datastream sinks to a storage and on particular failed
> > attempts I'd like to deadletter to a kafka topic.
> > Any suggestions?
> >
> > Thanks
> >
>