Re: Important (proposed) CEP changes for Flink 1.5.

[Vishal Santoshi]

That should be fine by us. We do not plan to do a SP and restore on our prod CEP pipe, when we move to 1.5 from 1.4. I do hope that all other pipes will remain back ward compatible ( 1.4 to 1.5 ).

On Wed, Feb 21, 2018 at 5:06 AM, Kostas Kloudas <[hidden email]> wrote:

Hi all,

Currently due to backwards compatibility there are some issues that seem to be affecting CEP users that use RocksDB. As an example you can see this:

https://issues.apache.org/jira/browse/FLINK-7756

We suspect that the issues have mainly to do with the way state was kept internally in previous versions (using java serialization) and got inherited to the newer ones.

Given this, we would like to introduce a revamped state serialization strategy which solves the sources of these problems and also reduces the size of the state kept at each checkpoint. The problem with this, is that the state format changes so already existing jobs will not be able to restart from a savepoint taken from a previous version of the library.

I am writing this email to see if there are any objections to merging these changes in the 1.5 release which is about to be released. I suggest to wait for responses until Friday 5pm UTC+1, and if there is non, then we merge. If there are any, then we have to see how to proceed.

You can try out the changes in this branch: https://github.com/aljoscha/flink/commits/fix-flink-cep-serialization and report here if you have any problems or suggestions.

Also feel free to forward this to other users, if I forgot someone.

Cheers,
Kostas

[Kostas Kloudas-2]

Hi Vishal,

Yes these changes will only affect CEP.

Cheers,

Kostas

On Feb 21, 2018, at 5:42 PM, Vishal Santoshi <[hidden email]> wrote:

That should be fine by us. We do not plan to do a SP and restore on our prod CEP pipe, when we move to 1.5 from 1.4. I do hope that all other pipes will remain back ward compatible ( 1.4 to 1.5 ).

On Wed, Feb 21, 2018 at 5:06 AM, Kostas Kloudas <[hidden email]> wrote:

Hi all,

Currently due to backwards compatibility there are some issues that seem to be affecting CEP users that use RocksDB. As an example you can see this:

https://issues.apache.org/jira/browse/FLINK-7756

We suspect that the issues have mainly to do with the way state was kept internally in previous versions (using java serialization) and got inherited to the newer ones.

Given this, we would like to introduce a revamped state serialization strategy which solves the sources of these problems and also reduces the size of the state kept at each checkpoint. The problem with this, is that the state format changes so already existing jobs will not be able to restart from a savepoint taken from a previous version of the library.

I am writing this email to see if there are any objections to merging these changes in the 1.5 release which is about to be released. I suggest to wait for responses until Friday 5pm UTC+1, and if there is non, then we merge. If there are any, then we have to see how to proceed.

You can try out the changes in this branch: https://github.com/aljoscha/flink/commits/fix-flink-cep-serialization and report here if you have any problems or suggestions.

Also feel free to forward this to other users, if I forgot someone.

Cheers,
Kostas