Is there a way we can specify operator ID for DDLs?

[Kevin Kwon]

For SQLs, I know that the operator ID assignment is not possible now since the query optimizer may not be backward compatible in each release

But are DDLs also affected by this?

for example,

CREATE TABLE mytable (

  id BIGINT,

  data STRING

) with (

  connector = 'kafka'

  ...

  id = 'mytable'

  name = 'mytable'

)

and we can save all related checkpoint data

[Danny Chan-2]

Hi Kevin Kwon ~

Do you want to customize only the source operator name or all the operator name in order for the state compatibility ?

State compatibility is an orthogonal topic and keep the operator name is one way to solve it.

Kevin Kwon <[hidden email]> 于2020年11月25日周三 上午1:11写道：

For SQLs, I know that the operator ID assignment is not possible now since the query optimizer may not be backward compatible in each release

But are DDLs also affected by this?

for example,

CREATE TABLE mytable (

  id BIGINT,

  data STRING

) with (

  connector = 'kafka'

  ...

  id = 'mytable'

  name = 'mytable'

)

and we can save all related checkpoint data

[Danny Chan-2]

SQL does not support that now. But i think your request is reasonable. AFAIK . SQL hints may be a way to configure such a per-operator thing. Would fire an issue first to see if we have an solution for the midterm.

Kevin Kwon <[hidden email]>于2020年11月25日 周三下午5:06写道：

I just want the source and sink operator compatibility. I use Kafka as source and I'd want to save the offsets through checkpoint

I know how to do with DataStream API but not with plain SQL DDL

On Wed, Nov 25, 2020, 3:09 AM Danny Chan <[hidden email]> wrote:

Hi Kevin Kwon ~

Do you want to customize only the source operator name or all the operator name in order for the state compatibility ?

State compatibility is an orthogonal topic and keep the operator name is one way to solve it.

Kevin Kwon <[hidden email]> 于2020年11月25日周三 上午1:11写道：

For SQLs, I know that the operator ID assignment is not possible now since the query optimizer may not be backward compatible in each release

But are DDLs also affected by this?

for example,

CREATE TABLE mytable (

  id BIGINT,

  data STRING

) with (

  connector = 'kafka'

  ...

  id = 'mytable'

  name = 'mytable'

)

and we can save all related checkpoint data

[Danny Chan-2]

Here is the issue https://issues.apache.org/jira/browse/FLINK-20368

Kevin Kwon <[hidden email]> 于2020年11月26日周四 上午8:50写道：

thanks alot :)

On Wed, Nov 25, 2020 at 3:26 PM Danny Chan <[hidden email]> wrote:

SQL does not support that now. But i think your request is reasonable. AFAIK . SQL hints may be a way to configure such a per-operator thing. Would fire an issue first to see if we have an solution for the midterm.

Kevin Kwon <[hidden email]>于2020年11月25日 周三下午5:06写道：

I just want the source and sink operator compatibility. I use Kafka as source and I'd want to save the offsets through checkpoint

I know how to do with DataStream API but not with plain SQL DDL

On Wed, Nov 25, 2020, 3:09 AM Danny Chan <[hidden email]> wrote:

Hi Kevin Kwon ~

Do you want to customize only the source operator name or all the operator name in order for the state compatibility ?

State compatibility is an orthogonal topic and keep the operator name is one way to solve it.

Kevin Kwon <[hidden email]> 于2020年11月25日周三 上午1:11写道：

For SQLs, I know that the operator ID assignment is not possible now since the query optimizer may not be backward compatible in each release

But are DDLs also affected by this?

for example,

CREATE TABLE mytable (

  id BIGINT,

  data STRING

) with (

  connector = 'kafka'

  ...

  id = 'mytable'

  name = 'mytable'

)

and we can save all related checkpoint data