Get stream of rejected data from Elasticsearch6 sink

[Nick Triller]

Hi all,

 

is it possible to further process data that could not be persisted by the Elasticsearch6 sink without breaking checkpointing?

As I understand, the onFailure callback can’t be used to forward rejected data into a separate stream.

 

I would like to extend the sink if this use case is not covered yet.

What would be a reasonable approach that matches Flink’s overall architecture?

 

As a new Flink user, my first intuition was to check if it is possible to create a side output from the Elasticsearch6 sink.

Does it make sense to support sink side outputs?

 

Thank you for your advice.

 

Regards,

Nick

[vino yang]

Hi Nick,

When you get the failed data, the logic for implementing the side output is similar to the logic for extending the ActionRequestFailureHandler#onFailure method to output the data to other places.

Thanks, vino.

Nick Triller <[hidden email]> 于2018年8月31日周五 下午9:08写道：

Hi all,

 

is it possible to further process data that could not be persisted by the Elasticsearch6 sink without breaking checkpointing?

As I understand, the onFailure callback can’t be used to forward rejected data into a separate stream.

 

I would like to extend the sink if this use case is not covered yet.

What would be a reasonable approach that matches Flink’s overall architecture?

 

As a new Flink user, my first intuition was to check if it is possible to create a side output from the Elasticsearch6 sink.

Does it make sense to support sink side outputs?

 

Thank you for your advice.

 

Regards,

Nick