[POLL] Dropping savepoint format compatibility for 1.1.x in the Flink 1.4.0 release

[Stephan Ewen]

Hi users!

Flink currently maintains backwards compatibility for savepoint formats, which means that savepoints taken with Flink version 1.1.x and 1.2.x can be resumed in Flink 1.3.x

We are discussing how many versions back to support. The proposition is the following:

   Suggestion: Flink 1.4.0 will be able to resume savepoints taken with version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x

The reason for that is that there is a lot of code mapping between the completely different legacy format (1.1.x, not re-scalable) and the key-group-oriented format (1.2.x onwards, re-scalable). It would greatly help the development of state and checkpointing features to drop that old code.

Please let us know if you have concerns about that.

Best,

Stephan

[Kanstantsin Kamkou]

Hi users!

I can't upgrade from 1.2.x to 1.3.x without code adaptations.
Upgrading from 1.(0|1).x to 1.2.x produces configuration mess.
Maybe you can discuss changing the release plan, speed it up a little
bit and use the major.minor.patch versions as advantage to organize
the release process more accurate. As the result, backwards
compatibility might be guarantied among a single major number only. It
also sounds like a new version will be 2.0.0. This way flink might get
more trust as I will see the complexity of changing the version
number.

> MINOR version when you add functionality in a backwards-compatible manner
That is exactly my expectation.

Best.


On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <[hidden email]> wrote:

> Hi users!
>
> Flink currently maintains backwards compatibility for savepoint formats,
> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
> resumed in Flink 1.3.x
>
> We are discussing how many versions back to support. The proposition is the
> following:
>
>    Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x
>
>
> The reason for that is that there is a lot of code mapping between the
> completely different legacy format (1.1.x, not re-scalable) and the
> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
> help the development of state and checkpointing features to drop that old
> code.
>
> Please let us know if you have concerns about that.
>
> Best,
> Stephan
>

[Günter Hipler]

+1 to drop Java 7 support

On 12.07.2017 16:43, Stephan Ewen wrote:

Hi users!

Flink currently maintains backwards compatibility for savepoint formats, which means that savepoints taken with Flink version 1.1.x and 1.2.x can be resumed in Flink 1.3.x

We are discussing how many versions back to support. The proposition is the following:

   Suggestion: Flink 1.4.0 will be able to resume savepoints taken with version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x

The reason for that is that there is a lot of code mapping between the completely different legacy format (1.1.x, not re-scalable) and the key-group-oriented format (1.2.x onwards, re-scalable). It would greatly help the development of state and checkpointing features to drop that old code.

Please let us know if you have concerns about that.

Best,

Stephan

[Stephan Ewen]

Hi Kanstantsin!

Looking at other mail thread, I think you refer to the CEP library API, correct?

The CEP library has not been labeled as stable yet. The stability guarantees that we introduced refer to the APIs marked with the `@Public` interface.

The CEP library was still too much in change/improvement mode to freeze it already and mark it as stable.

Have you seen parts marked as `@Public` where you had to adjust code?

Best,

Stephan

On Wed, Jul 12, 2017 at 6:12 PM, Kanstantsin Kamkou <[hidden email]> wrote:

Hi users!

I can't upgrade from 1.2.x to 1.3.x without code adaptations.
Upgrading from 1.(0|1).x to 1.2.x produces configuration mess.
Maybe you can discuss changing the release plan, speed it up a little
bit and use the major.minor.patch versions as advantage to organize
the release process more accurate. As the result, backwards
compatibility might be guarantied among a single major number only. It
also sounds like a new version will be 2.0.0. This way flink might get
more trust as I will see the complexity of changing the version
number.

> MINOR version when you add functionality in a backwards-compatible manner
That is exactly my expectation.

Best.

On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <[hidden email]> wrote:
> Hi users!
>
> Flink currently maintains backwards compatibility for savepoint formats,
> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
> resumed in Flink 1.3.x
>
> We are discussing how many versions back to support. The proposition is the
> following:
>
>    Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x
>
>
> The reason for that is that there is a lot of code mapping between the
> completely different legacy format (1.1.x, not re-scalable) and the
> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
> help the development of state and checkpointing features to drop that old
> code.
>
> Please let us know if you have concerns about that.
>
> Best,
> Stephan
>

[Stephan Ewen]

Seems like no one raised a concern so far about dropping the savepoint format compatibility for 1.1 in 1.4.

Leaving this thread open for some more days, but from the sentiment, it seems like we should go ahead?

On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <[hidden email]> wrote:

Hi users!

Flink currently maintains backwards compatibility for savepoint formats, which means that savepoints taken with Flink version 1.1.x and 1.2.x can be resumed in Flink 1.3.x

We are discussing how many versions back to support. The proposition is the following:

   Suggestion: Flink 1.4.0 will be able to resume savepoints taken with version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x

The reason for that is that there is a lot of code mapping between the completely different legacy format (1.1.x, not re-scalable) and the key-group-oriented format (1.2.x onwards, re-scalable). It would greatly help the development of state and checkpointing features to drop that old code.

Please let us know if you have concerns about that.

Best,

Stephan

[Ufuk Celebi]

On Fri, Jul 28, 2017 at 4:03 PM, Stephan Ewen <[hidden email]> wrote:
> Seems like no one raised a concern so far about dropping the savepoint
> format compatibility for 1.1 in 1.4.
>
> Leaving this thread open for some more days, but from the sentiment, it
> seems like we should go ahead?

+1

[Stefan Richter]

+1

Am 28.07.2017 um 16:03 schrieb Stephan Ewen <[hidden email]>:

Seems like no one raised a concern so far about dropping the savepoint format compatibility for 1.1 in 1.4.

Leaving this thread open for some more days, but from the sentiment, it seems like we should go ahead?

On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <[hidden email]> wrote:

Hi users!

Flink currently maintains backwards compatibility for savepoint formats, which means that savepoints taken with Flink version 1.1.x and 1.2.x can be resumed in Flink 1.3.x

We are discussing how many versions back to support. The proposition is the following:

   Suggestion: Flink 1.4.0 will be able to resume savepoints taken with version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x

The reason for that is that there is a lot of code mapping between the completely different legacy format (1.1.x, not re-scalable) and the key-group-oriented format (1.2.x onwards, re-scalable). It would greatly help the development of state and checkpointing features to drop that old code.

Please let us know if you have concerns about that.

Best,

Stephan

[Till Rohrmann]

+1

On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <[hidden email]> wrote:

+1

Am 28.07.2017 um 16:03 schrieb Stephan Ewen <[hidden email]>:

Seems like no one raised a concern so far about dropping the savepoint format compatibility for 1.1 in 1.4.

Leaving this thread open for some more days, but from the sentiment, it seems like we should go ahead?

On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <[hidden email]> wrote:

Hi users!

Flink currently maintains backwards compatibility for savepoint formats, which means that savepoints taken with Flink version 1.1.x and 1.2.x can be resumed in Flink 1.3.x

We are discussing how many versions back to support. The proposition is the following:

   Suggestion: Flink 1.4.0 will be able to resume savepoints taken with version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x

The reason for that is that there is a lot of code mapping between the completely different legacy format (1.1.x, not re-scalable) and the key-group-oriented format (1.2.x onwards, re-scalable). It would greatly help the development of state and checkpointing features to drop that old code.

Please let us know if you have concerns about that.

Best,

Stephan

[Kostas Kloudas]

+1

> On Aug 2, 2017, at 3:16 PM, Till Rohrmann <[hidden email]> wrote:
>
> +1
>
> On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <[hidden email]>
> wrote:
>
>> +1
>>
>> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <[hidden email]>:
>>
>> Seems like no one raised a concern so far about dropping the savepoint
>> format compatibility for 1.1 in 1.4.
>>
>> Leaving this thread open for some more days, but from the sentiment, it
>> seems like we should go ahead?
>>
>> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <[hidden email]> wrote:
>>
>>> Hi users!
>>>
>>> Flink currently maintains backwards compatibility for savepoint formats,
>>> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
>>> resumed in Flink 1.3.x
>>>
>>> We are discussing how many versions back to support. The proposition is
>>> the following:
>>>
>>> *   Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
>>> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>>>
>>>
>>> The reason for that is that there is a lot of code mapping between the
>>> completely different legacy format (1.1.x, not re-scalable) and the
>>> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
>>> help the development of state and checkpointing features to drop that old
>>> code.
>>>
>>> Please let us know if you have concerns about that.
>>>
>>> Best,
>>> Stephan
>>>
>>>
>>
>>