CEP operator in SQL pattern match does not clear it's state

[Muhammad Hakim]

Hi All,

We've been experimenting with `MATCH_RECOGNIZE` clause on Flink 1.8. We tried to have a pattern match within one hour. According to https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html#time-constraint , using `WITHIN` will control the memory usage. We've been running queries using CEP for almost one week and memory usage is not going down. Are we missing something?

[Dawid Wysakowicz-2]

Hi,

Could you give a bit more details about your program? What is the query
you are using? What is the incoming data?

The "within" clause ensures that all partial matches that were initiated
at time t, are cleared when an event with timestamp (t + w) appears,
where w is the interval specified in "within" clause. The reason why it
is a good practice to use it for memory control is that it ensures that
there will be no partial matches dangling forever.

If the rate of incoming events and the probability of initiating new
partial matches is constant in the input stream I would not expect the
memory consumption to go down, but it should saturate after some time
and remain rather constant, but should not grow infinitely.

I hope it helps.

Best,

Dawid

On 01/10/2019 09:08, Muhammad Hakim wrote:
> Hi All,
> We've been experimenting with `MATCH_RECOGNIZE` clause on Flink 1.8.
> We tried to have a pattern match within one hour. According
> to https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html#time-constraint ,
> using `WITHIN` will control the memory usage. We've been running
> queries using CEP for almost one week and memory usage is not going
> down. Are we missing something?

[Theo]

Hi Muhammad,

With "not going done" you mean that it's still growing or that it's constant? In case of it being constant, that's pretty much what is expected, right? If it's still growing, the only reason I could come up with: Do you work in event time and have watermarks properly assigned with a boundedTimestampExtractor or ascendingTimestampExtractor?

Best regards

Theo

---

Von: "Muhammad Hakim" <[hidden email]>
An: "user" <[hidden email]>
Gesendet: Dienstag, 1. Oktober 2019 09:08:22
Betreff: CEP operator in SQL pattern match does not clear it's state

Hi All,

We've been experimenting with `MATCH_RECOGNIZE` clause on Flink 1.8. We tried to have a pattern match within one hour. According to https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html#time-constraint , using `WITHIN` will control the memory usage. We've been running queries using CEP for almost one week and memory usage is not going down. Are we missing something?