Table API: get Schema from TableSchema, TypeInformation<?>[] or Avro

[françois lacombe]

Hi all,

Today I'm looking into derivating an Avro schema json string into a Schema object.

In the overview of https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html Avro is used as a format and never as a schema.

This was a topic in JIRA-9813

I can get a TableSchema with TableSchema schema = TableSchema.fromTypeInfo(AvroSchemaConverter.convertToTypeInfo(sch_csv.toString())); but I can't use it with BatchTableDescriptor.withSchema().

How can I get a Schema from TableSchema, TypeInformation<?>[] or even Avro json string?

A little bridge is missing between TableSchema and org.apache.flink.table.descriptors.Schema it seems.

Thanks in advance for any useful hint

François

[Timo Walther]

Hi,

thanks for your feedback. I agree that the the current interfaces are
not flexible enough to fit to every use case. The unified connector API
is a a very recent feature that still needs some polishing. I'm working
on a design document to improve the situation there.

For now, you can simply implement some utitilty method that just
iterates over column names and types of TableSchema and calls
`schema.field(name, type)`

I hope this helps.

Regards,
Timo

Am 31.08.18 um 07:40 schrieb françois lacombe:

> Hi all,
>
> Today I'm looking into derivating an Avro schema json string into a
> Schema object.
> In the overview of
> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html 
> Avro is used as a format and never as a schema.
>
> This was a topic in JIRA-9813
> I can get a TableSchema with TableSchema schema =
> TableSchema.fromTypeInfo(AvroSchemaConverter.convertToTypeInfo(sch_csv.toString()));
> but I can't use it with BatchTableDescriptor.withSchema().
>
> How can I get a Schema from TableSchema, TypeInformation<?>[] or even
> Avro json string?
> A little bridge is missing between TableSchema and
> org.apache.flink.table.descriptors.Schema it seems.
>
> Thanks in advance for any useful hint
>
> François

[françois lacombe]

Hi Timo

Yes it helps, thank you.

I'll start building such an utility method. Are you interested to get the source?

According to mapping here : https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html#apache-avro-format

Is there any way to get corresponding TypeInformation<?> of an Avro type or should I hard code a List<String, TypeInformation<?>>?

All the best

François

2018-08-31 8:12 GMT+02:00 Timo Walther <[hidden email]>:

Hi,

thanks for your feedback. I agree that the the current interfaces are not flexible enough to fit to every use case. The unified connector API is a a very recent feature that still needs some polishing. I'm working on a design document to improve the situation there.

For now, you can simply implement some utitilty method that just iterates over column names and types of TableSchema and calls `schema.field(name, type)`

I hope this helps.

Regards,
Timo

Am 31.08.18 um 07:40 schrieb françois lacombe:

Hi all,

Today I'm looking into derivating an Avro schema json string into a Schema object.
In the overview of https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html Avro is used as a format and never as a schema.

This was a topic in JIRA-9813
I can get a TableSchema with TableSchema schema = TableSchema.fromTypeInfo(AvroSchemaConverter.convertToTypeInfo(sch_csv.toString())); but I can't use it with BatchTableDescriptor.withSchema().

How can I get a Schema from TableSchema, TypeInformation<?>[] or even Avro json string?
A little bridge is missing between TableSchema and org.apache.flink.table.descriptors.Schema it seems.

Thanks in advance for any useful hint

François

[Timo Walther]

Thanks for your response. I think we won't need this utility in the near future. As mentioned, I'm working on a design document that allows for better abstraction. I think I will publish it next week.

Regards,
Timo


Am 31.08.18 um 08:36 schrieb françois lacombe:

Hi Timo

Yes it helps, thank you.

I'll start building such an utility method. Are you interested to get the source?

According to mapping here : https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html#apache-avro-format

Is there any way to get corresponding TypeInformation<?> of an Avro type or should I hard code a List<String, TypeInformation<?>>?

All the best

François

2018-08-31 8:12 GMT+02:00 Timo Walther <[hidden email]>:

Hi,

thanks for your feedback. I agree that the the current interfaces are not flexible enough to fit to every use case. The unified connector API is a a very recent feature that still needs some polishing. I'm working on a design document to improve the situation there.

For now, you can simply implement some utitilty method that just iterates over column names and types of TableSchema and calls `schema.field(name, type)`

I hope this helps.

Regards,
Timo

Am 31.08.18 um 07:40 schrieb françois lacombe:

Hi all,

Today I'm looking into derivating an Avro schema json string into a Schema object.
In the overview of https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html Avro is used as a format and never as a schema.

This was a topic in JIRA-9813
I can get a TableSchema with TableSchema schema = TableSchema.fromTypeInfo(AvroSchemaConverter.convertToTypeInfo(sch_csv.toString())); but I can't use it with BatchTableDescriptor.withSchema().

How can I get a Schema from TableSchema, TypeInformation<?>[] or even Avro json string?
A little bridge is missing between TableSchema and org.apache.flink.table.descriptors.Schema it seems.

Thanks in advance for any useful hint

François

[françois lacombe]

Ok, looking forward reading your document.

All the best

François

2018-08-31 9:09 GMT+02:00 Timo Walther <[hidden email]>:

Thanks for your response. I think we won't need this utility in the near future. As mentioned, I'm working on a design document that allows for better abstraction. I think I will publish it next week.

Regards,
Timo


Am 31.08.18 um 08:36 schrieb françois lacombe:

Hi Timo

Yes it helps, thank you.

I'll start building such an utility method. Are you interested to get the source?

According to mapping here : https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html#apache-avro-format

Is there any way to get corresponding TypeInformation<?> of an Avro type or should I hard code a List<String, TypeInformation<?>>?

All the best

François

2018-08-31 8:12 GMT+02:00 Timo Walther <[hidden email]>:

Hi,

thanks for your feedback. I agree that the the current interfaces are not flexible enough to fit to every use case. The unified connector API is a a very recent feature that still needs some polishing. I'm working on a design document to improve the situation there.

For now, you can simply implement some utitilty method that just iterates over column names and types of TableSchema and calls `schema.field(name, type)`

I hope this helps.

Regards,
Timo

Am 31.08.18 um 07:40 schrieb françois lacombe:

Hi all,

Today I'm looking into derivating an Avro schema json string into a Schema object.
In the overview of https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/connect.html Avro is used as a format and never as a schema.

This was a topic in JIRA-9813
I can get a TableSchema with TableSchema schema = TableSchema.fromTypeInfo(AvroSchemaConverter.convertToTypeInfo(sch_csv.toString())); but I can't use it with BatchTableDescriptor.withSchema().

How can I get a Schema from TableSchema, TypeInformation<?>[] or even Avro json string?
A little bridge is missing between TableSchema and org.apache.flink.table.descriptors.Schema it seems.

Thanks in advance for any useful hint

François