Emit event to kafka when finish sink

[Antonio Manzano]

Hello guys, 

i would like to know if there is any possibility to emit an event when a sink has finished.
To put it in context, I have a simple ETL (streaming bounded) that reads data from a database, maps, and inserts into another database. Once I finish inserting the data I want to issue an event to kafka to notify another process.

I can think of some solutions, but I am not convinced which one is the best option:

• dividing the stream into two paths and with a session window, in onTimer,  check if the last record is already in the database
• Implement custom sink
• ...

Any suggestions?

Thanks!

--

[Dawid Wysakowicz-2]

Hi,

I am not sure if I understand your first solution, but it sounds rather complicated.

I think implementing a custom operator could be a valid approach. You would have to make sure it is run with parallelism of 1. You could additionally implement a BoundedOneInput interface and notify the external process in the endOfInput method. This is so also quite involved solution.

Another idea is you could register a JobListener in the StreamExecutionEnvironment. There you can execute some code based on the status of the whole job.

Best,

Dawid

On 15/09/2020 11:00, Antonio Manzano wrote:

Hello guys, 

i would like to know if there is any possibility to emit an event when a sink has finished.
To put it in context, I have a simple ETL (streaming bounded) that reads data from a database, maps, and inserts into another database. Once I finish inserting the data I want to issue an event to kafka to notify another process.

I can think of some solutions, but I am not convinced which one is the best option:

• dividing the stream into two paths and with a session window, in onTimer,  check if the last record is already in the database
• Implement custom sink
• ...

Any suggestions?

Thanks!

--