Re: Flink CEP not emitting timed out events properly

Posted by Biplob Biswas on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-CEP-not-emitting-timed-out-events-properly-tp13794p13846.html

Hi Kostas,

Implementing my custom timestamp assigned made em realise a problem which we have in our architecture you may say.

Any inputs would be really appreciated.

So, for now, we are reading from 4 different kafka topics, and we have a flow similar to something like this:

Event 1(on topic t1) -generates-> Event 2(on topic t2)
Event 1(on topic t1) -generates-> Event 3(on topic t3) -generates-> Event 4 (on topic t4)

As I am reading from all these topics, there is an overlap of the various events with the various timestamp, so when I am printing the list of events, I can see events with higher timestamps as well which I assumed is affecting the watermarks.

But when I added logs to the extract timestamp method, I can see that there are multiple different watermarks which are maintained.

Could you or anyone else help me in understanding, how watermarks work in case of reading from different topics simultaneously?

I am pretty sure that my lost events are due to the issues with the way watermarking works on events from various topics simultaneously.

Thanks & Regards
Biplob