(DEPRECATED) Apache Flink User Mailing List archive.

State Schema Evolution within SQL API

Classic

List

Threaded

5 messages Options

Jan Oelschlegel

State Schema Evolution within SQL API

Hi at all,

i would like to know how far a state schema evolution is possible by using SQL API of Flink. Which query changes can I do without disrupting the schema of my savepoint?

In the documentation is, only for the DataStream API , written what are the do’s and don’ts regarding a safe schema evolution. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html

Best,

Jan

HINWEIS: Dies ist eine vertrauliche Nachricht und nur für den Adressaten bestimmt. Es ist nicht erlaubt, diese Nachricht zu kopieren oder Dritten zugänglich zu machen. Sollten Sie diese Nachricht irrtümlich erhalten haben, bitte ich um Ihre Mitteilung per E-Mail oder unter der oben angegebenen Telefonnummer.

Dawid Wysakowicz-2

Re: State Schema Evolution within SQL API

Hi Jan,

As of now Flink does not give any guarantees for Table/SQL API savepoint compatibility if you change the query or Flink version. Flink Table/SQL API uses an optimizer that can apply different optimizations or operations reordering based on the queried fields or computations that can result in a completely different physical plan.

Generally speaking you should be fine when adding/removing fields in a projection. I'd say it is the only somewhat safe change, but it is not guaranteed in all cases nevertheless.

Best,

Dawid

On 01/03/2021 17:41, Jan Oelschlegel wrote:

Hi at all,

i would like to know how far a state schema evolution is possible by using SQL API of Flink. Which query changes can I do without disrupting the schema of my savepoint?

In the documentation is, only for the DataStream API , written what are the do’s and don’ts regarding a safe schema evolution. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html

Best,

Jan

HINWEIS: Dies ist eine vertrauliche Nachricht und nur für den Adressaten bestimmt. Es ist nicht erlaubt, diese Nachricht zu kopieren oder Dritten zugänglich zu machen. Sollten Sie diese Nachricht irrtümlich erhalten haben, bitte ich um Ihre Mitteilung per E-Mail oder unter der oben angegebenen Telefonnummer.

OpenPGP_signature (855 bytes) Download Attachment

XU Qinghui

Re: State Schema Evolution within SQL API

Hello Dawid

I'm interested in this discussion because I'm currently trying to upgrade flink from 1.9 to 1.12 for a bunch of sql jobs running in production. From what you said, this seems to be a risky operation which might corrupt the states. Is there any recommendation to mitigate the risk?

It would be pretty hard to maintain sql jobs on prod if each upgrade undergoes such risk.

Best regards,

Qinghui

Le jeu. 4 mars 2021 à 09:50, Dawid Wysakowicz <[hidden email]> a écrit :

Hi Jan,

As of now Flink does not give any guarantees for Table/SQL API savepoint compatibility if you change the query or Flink version. Flink Table/SQL API uses an optimizer that can apply different optimizations or operations reordering based on the queried fields or computations that can result in a completely different physical plan.

Generally speaking you should be fine when adding/removing fields in a projection. I'd say it is the only somewhat safe change, but it is not guaranteed in all cases nevertheless.

Best,

Dawid

On 01/03/2021 17:41, Jan Oelschlegel wrote:

Hi at all,

i would like to know how far a state schema evolution is possible by using SQL API of Flink. Which query changes can I do without disrupting the schema of my savepoint?

In the documentation is, only for the DataStream API , written what are the do’s and don’ts regarding a safe schema evolution. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html

Best,

Jan

HINWEIS: Dies ist eine vertrauliche Nachricht und nur für den Adressaten bestimmt. Es ist nicht erlaubt, diese Nachricht zu kopieren oder Dritten zugänglich zu machen. Sollten Sie diese Nachricht irrtümlich erhalten haben, bitte ich um Ihre Mitteilung per E-Mail oder unter der oben angegebenen Telefonnummer.

Jan Oelschlegel

AW: State Schema Evolution within SQL API

Hi Qinghui

Maybe this strategy with evolving an application without the need for state restore could help you [1]

[1] https://docs.ververica.com/v2.3/user_guide/sql_development/sql_scripts.html#sql-script-changes

Best,

Jan

Von: XU Qinghui <[hidden email]>
Gesendet: Donnerstag, 4. März 2021 10:18
An: Dawid Wysakowicz <[hidden email]>
Cc: Jan Oelschlegel <[hidden email]>; user <[hidden email]>
Betreff: Re: State Schema Evolution within SQL API

Hello Dawid

It would be pretty hard to maintain sql jobs on prod if each upgrade undergoes such risk.

Best regards,

Qinghui

Le jeu. 4 mars 2021 à 09:50, Dawid Wysakowicz <[hidden email]> a écrit :

Hi Jan,

As of now Flink does not give any guarantees for Table/SQL API savepoint compatibility if you change the query or Flink version. Flink Table/SQL API uses an optimizer that can apply different optimizations or operations reordering based on the queried fields or computations that can result in a completely different physical plan.

Generally speaking you should be fine when adding/removing fields in a projection. I'd say it is the only somewhat safe change, but it is not guaranteed in all cases nevertheless.

Best,

Dawid

On 01/03/2021 17:41, Jan Oelschlegel wrote:

Hi at all,

i would like to know how far a state schema evolution is possible by using SQL API of Flink. Which query changes can I do without disrupting the schema of my savepoint?

In the documentation is, only for the DataStream API , written what are the do’s and don’ts regarding a safe schema evolution. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html

Best,

Jan

HINWEIS: Dies ist eine vertrauliche Nachricht und nur für den Adressaten bestimmt. Es ist nicht erlaubt, diese Nachricht zu kopieren oder Dritten zugänglich zu machen. Sollten Sie diese Nachricht irrtümlich erhalten haben, bitte ich um Ihre Mitteilung per E-Mail oder unter der oben angegebenen Telefonnummer.

Jan Oelschlegel

AW: State Schema Evolution within SQL API

I think simply changing the parallelism or upgrading the underlying Flink version in a cluster should be no problem.

But you want to change your queries, right ?

Best,

Jan

Von: Jan Oelschlegel <[hidden email]>
Gesendet: Donnerstag, 4. März 2021 12:26
An: XU Qinghui <[hidden email]>; Dawid Wysakowicz <[hidden email]>
Cc: user <[hidden email]>
Betreff: AW: State Schema Evolution within SQL API

Hi Qinghui

Maybe this strategy with evolving an application without the need for state restore could help you [1]

[1] https://docs.ververica.com/v2.3/user_guide/sql_development/sql_scripts.html#sql-script-changes

Best,

Jan

Hello Dawid

It would be pretty hard to maintain sql jobs on prod if each upgrade undergoes such risk.

Best regards,

Qinghui

Le jeu. 4 mars 2021 à 09:50, Dawid Wysakowicz <[hidden email]> a écrit :

Hi Jan,

As of now Flink does not give any guarantees for Table/SQL API savepoint compatibility if you change the query or Flink version. Flink Table/SQL API uses an optimizer that can apply different optimizations or operations reordering based on the queried fields or computations that can result in a completely different physical plan.

Generally speaking you should be fine when adding/removing fields in a projection. I'd say it is the only somewhat safe change, but it is not guaranteed in all cases nevertheless.

Best,

Dawid

On 01/03/2021 17:41, Jan Oelschlegel wrote:

Hi at all,

i would like to know how far a state schema evolution is possible by using SQL API of Flink. Which query changes can I do without disrupting the schema of my savepoint?

In the documentation is, only for the DataStream API , written what are the do’s and don’ts regarding a safe schema evolution. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html

Best,

Jan

HINWEIS: Dies ist eine vertrauliche Nachricht und nur für den Adressaten bestimmt. Es ist nicht erlaubt, diese Nachricht zu kopieren oder Dritten zugänglich zu machen. Sollten Sie diese Nachricht irrtümlich erhalten haben, bitte ich um Ihre Mitteilung per E-Mail oder unter der oben angegebenen Telefonnummer.