Read data from Oracle using Flink SQL API

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Read data from Oracle using Flink SQL API

Flavio Pompermaier
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio
Reply | Threaded
Open this post in threaded view
|

Re: Read data from Oracle using Flink SQL API

Jark Wu-3
Hi Flavio,

If you want to adjust the writing statement for Oracle, you can implement the JDBCDialect for Oracle, and pass to the JDBCUpsertTableSink when constructing via `JDBCOptions.Builder#setDialect`. In this way, you don't need to recompile the source code of flink-jdbc.

Best,
Jark

On Fri, 31 Jan 2020 at 19:28, Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio
Reply | Threaded
Open this post in threaded view
|

Re: Read data from Oracle using Flink SQL API

Jingsong Li
Hi Flavio,

You can use `JDBCTableSource`, and register it from
 TableEnvionment.registerTableSource, you need provide
 a OracleDialect, maybe just implement `canHandle` and
 `defaultDriverName` is OK.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 2:42 PM Jark Wu <[hidden email]> wrote:
Hi Flavio,

If you want to adjust the writing statement for Oracle, you can implement the JDBCDialect for Oracle, and pass to the JDBCUpsertTableSink when constructing via `JDBCOptions.Builder#setDialect`. In this way, you don't need to recompile the source code of flink-jdbc.

Best,
Jark

On Fri, 31 Jan 2020 at 19:28, Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio


--
Best, Jingsong Lee
Reply | Threaded
Open this post in threaded view
|

Re: Read data from Oracle using Flink SQL API

Flavio Pompermaier
Ok thanks for this info! Maybe this could be added to the documentation..what do you think?

Il Dom 2 Feb 2020, 08:37 Jingsong Li <[hidden email]> ha scritto:
Hi Flavio,

You can use `JDBCTableSource`, and register it from
 TableEnvionment.registerTableSource, you need provide
 a OracleDialect, maybe just implement `canHandle` and
 `defaultDriverName` is OK.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 2:42 PM Jark Wu <[hidden email]> wrote:
Hi Flavio,

If you want to adjust the writing statement for Oracle, you can implement the JDBCDialect for Oracle, and pass to the JDBCUpsertTableSink when constructing via `JDBCOptions.Builder#setDialect`. In this way, you don't need to recompile the source code of flink-jdbc.

Best,
Jark

On Fri, 31 Jan 2020 at 19:28, Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio


--
Best, Jingsong Lee
Reply | Threaded
Open this post in threaded view
|

Re: Read data from Oracle using Flink SQL API

Jingsong Li
Yes, And I think we should add OracleDialect,SqlServerDialect,DB2Dialect support too.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 5:53 PM Flavio Pompermaier <[hidden email]> wrote:
Ok thanks for this info! Maybe this could be added to the documentation..what do you think?

Il Dom 2 Feb 2020, 08:37 Jingsong Li <[hidden email]> ha scritto:
Hi Flavio,

You can use `JDBCTableSource`, and register it from
 TableEnvionment.registerTableSource, you need provide
 a OracleDialect, maybe just implement `canHandle` and
 `defaultDriverName` is OK.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 2:42 PM Jark Wu <[hidden email]> wrote:
Hi Flavio,

If you want to adjust the writing statement for Oracle, you can implement the JDBCDialect for Oracle, and pass to the JDBCUpsertTableSink when constructing via `JDBCOptions.Builder#setDialect`. In this way, you don't need to recompile the source code of flink-jdbc.

Best,
Jark

On Fri, 31 Jan 2020 at 19:28, Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio


--
Best, Jingsong Lee


--
Best, Jingsong Lee
Reply | Threaded
Open this post in threaded view
|

Re: Read data from Oracle using Flink SQL API

phoenixjiangnan
Hi Flavio,

+1 for adding Oracle (potentially more dbms like SqlServer, etc) to flink-jdbc. Would you mind open a parent ticket and some subtasks, each one for one to-be-added dbms you've thought of?


On Sun, Feb 2, 2020 at 10:11 PM Jingsong Li <[hidden email]> wrote:
Yes, And I think we should add OracleDialect,SqlServerDialect,DB2Dialect support too.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 5:53 PM Flavio Pompermaier <[hidden email]> wrote:
Ok thanks for this info! Maybe this could be added to the documentation..what do you think?

Il Dom 2 Feb 2020, 08:37 Jingsong Li <[hidden email]> ha scritto:
Hi Flavio,

You can use `JDBCTableSource`, and register it from
 TableEnvionment.registerTableSource, you need provide
 a OracleDialect, maybe just implement `canHandle` and
 `defaultDriverName` is OK.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 2:42 PM Jark Wu <[hidden email]> wrote:
Hi Flavio,

If you want to adjust the writing statement for Oracle, you can implement the JDBCDialect for Oracle, and pass to the JDBCUpsertTableSink when constructing via `JDBCOptions.Builder#setDialect`. In this way, you don't need to recompile the source code of flink-jdbc.

Best,
Jark

On Fri, 31 Jan 2020 at 19:28, Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio


--
Best, Jingsong Lee


--
Best, Jingsong Lee
Reply | Threaded
Open this post in threaded view
|

Re: Read data from Oracle using Flink SQL API

Jingsong Li
Hi Bowen,


Best,
Jingsong Lee

On Thu, Feb 6, 2020 at 12:57 PM Bowen Li <[hidden email]> wrote:
Hi Flavio,

+1 for adding Oracle (potentially more dbms like SqlServer, etc) to flink-jdbc. Would you mind open a parent ticket and some subtasks, each one for one to-be-added dbms you've thought of?


On Sun, Feb 2, 2020 at 10:11 PM Jingsong Li <[hidden email]> wrote:
Yes, And I think we should add OracleDialect,SqlServerDialect,DB2Dialect support too.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 5:53 PM Flavio Pompermaier <[hidden email]> wrote:
Ok thanks for this info! Maybe this could be added to the documentation..what do you think?

Il Dom 2 Feb 2020, 08:37 Jingsong Li <[hidden email]> ha scritto:
Hi Flavio,

You can use `JDBCTableSource`, and register it from
 TableEnvionment.registerTableSource, you need provide
 a OracleDialect, maybe just implement `canHandle` and
 `defaultDriverName` is OK.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 2:42 PM Jark Wu <[hidden email]> wrote:
Hi Flavio,

If you want to adjust the writing statement for Oracle, you can implement the JDBCDialect for Oracle, and pass to the JDBCUpsertTableSink when constructing via `JDBCOptions.Builder#setDialect`. In this way, you don't need to recompile the source code of flink-jdbc.

Best,
Jark

On Fri, 31 Jan 2020 at 19:28, Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio


--
Best, Jingsong Lee


--
Best, Jingsong Lee


--
Best, Jingsong Lee
Reply | Threaded
Open this post in threaded view
|

Re: Read data from Oracle using Flink SQL API

Flavio Pompermaier
Should I open a JIRA to better document how to connect to an RDBMS that has no "out-of-the-box" dialect?
I.e. Pass the dialect impl in the TableEnvionment.registerTableSource?

On Thu, Feb 6, 2020 at 10:53 AM Jingsong Li <[hidden email]> wrote:
Hi Bowen,


Best,
Jingsong Lee

On Thu, Feb 6, 2020 at 12:57 PM Bowen Li <[hidden email]> wrote:
Hi Flavio,

+1 for adding Oracle (potentially more dbms like SqlServer, etc) to flink-jdbc. Would you mind open a parent ticket and some subtasks, each one for one to-be-added dbms you've thought of?


On Sun, Feb 2, 2020 at 10:11 PM Jingsong Li <[hidden email]> wrote:
Yes, And I think we should add OracleDialect,SqlServerDialect,DB2Dialect support too.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 5:53 PM Flavio Pompermaier <[hidden email]> wrote:
Ok thanks for this info! Maybe this could be added to the documentation..what do you think?

Il Dom 2 Feb 2020, 08:37 Jingsong Li <[hidden email]> ha scritto:
Hi Flavio,

You can use `JDBCTableSource`, and register it from
 TableEnvionment.registerTableSource, you need provide
 a OracleDialect, maybe just implement `canHandle` and
 `defaultDriverName` is OK.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 2:42 PM Jark Wu <[hidden email]> wrote:
Hi Flavio,

If you want to adjust the writing statement for Oracle, you can implement the JDBCDialect for Oracle, and pass to the JDBCUpsertTableSink when constructing via `JDBCOptions.Builder#setDialect`. In this way, you don't need to recompile the source code of flink-jdbc.

Best,
Jark

On Fri, 31 Jan 2020 at 19:28, Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio


--
Best, Jingsong Lee


--
Best, Jingsong Lee


--
Best, Jingsong Lee


--
Flavio Pompermaier
Development Department

OKKAM S.r.l.
Tel. +(39) 0461 041809
Reply | Threaded
Open this post in threaded view
|

Re: Read data from Oracle using Flink SQL API

Jingsong Li
Hi Flavio,

Instead of "TableEnvionment.registerTableSource", we want to drop it in 1.10.

I think you can create a JIRA to support pass dialect through JDBCTableFactory. We should support it. 

What do you think?

Best,
Jingsong Lee 

On Thu, Feb 6, 2020 at 6:01 PM Flavio Pompermaier <[hidden email]> wrote:
Should I open a JIRA to better document how to connect to an RDBMS that has no "out-of-the-box" dialect?
I.e. Pass the dialect impl in the TableEnvionment.registerTableSource?

On Thu, Feb 6, 2020 at 10:53 AM Jingsong Li <[hidden email]> wrote:
Hi Bowen,


Best,
Jingsong Lee

On Thu, Feb 6, 2020 at 12:57 PM Bowen Li <[hidden email]> wrote:
Hi Flavio,

+1 for adding Oracle (potentially more dbms like SqlServer, etc) to flink-jdbc. Would you mind open a parent ticket and some subtasks, each one for one to-be-added dbms you've thought of?


On Sun, Feb 2, 2020 at 10:11 PM Jingsong Li <[hidden email]> wrote:
Yes, And I think we should add OracleDialect,SqlServerDialect,DB2Dialect support too.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 5:53 PM Flavio Pompermaier <[hidden email]> wrote:
Ok thanks for this info! Maybe this could be added to the documentation..what do you think?

Il Dom 2 Feb 2020, 08:37 Jingsong Li <[hidden email]> ha scritto:
Hi Flavio,

You can use `JDBCTableSource`, and register it from
 TableEnvionment.registerTableSource, you need provide
 a OracleDialect, maybe just implement `canHandle` and
 `defaultDriverName` is OK.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 2:42 PM Jark Wu <[hidden email]> wrote:
Hi Flavio,

If you want to adjust the writing statement for Oracle, you can implement the JDBCDialect for Oracle, and pass to the JDBCUpsertTableSink when constructing via `JDBCOptions.Builder#setDialect`. In this way, you don't need to recompile the source code of flink-jdbc.

Best,
Jark

On Fri, 31 Jan 2020 at 19:28, Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio


--
Best, Jingsong Lee


--
Best, Jingsong Lee


--
Best, Jingsong Lee


--
Flavio Pompermaier
Development Department

OKKAM S.r.l.
Tel. +(39) 0461 041809


--
Best, Jingsong Lee
Reply | Threaded
Open this post in threaded view
|

Re: Read data from Oracle using Flink SQL API

Flavio Pompermaier
Ok to me, I just want this to be documented in at https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/connect.html#jdbc-connector. Do I need to open another JIRA for that?

On Thu, Feb 6, 2020 at 11:04 AM Jingsong Li <[hidden email]> wrote:
Hi Flavio,

Instead of "TableEnvionment.registerTableSource", we want to drop it in 1.10.

I think you can create a JIRA to support pass dialect through JDBCTableFactory. We should support it. 

What do you think?

Best,
Jingsong Lee 

On Thu, Feb 6, 2020 at 6:01 PM Flavio Pompermaier <[hidden email]> wrote:
Should I open a JIRA to better document how to connect to an RDBMS that has no "out-of-the-box" dialect?
I.e. Pass the dialect impl in the TableEnvionment.registerTableSource?

On Thu, Feb 6, 2020 at 10:53 AM Jingsong Li <[hidden email]> wrote:
Hi Bowen,


Best,
Jingsong Lee

On Thu, Feb 6, 2020 at 12:57 PM Bowen Li <[hidden email]> wrote:
Hi Flavio,

+1 for adding Oracle (potentially more dbms like SqlServer, etc) to flink-jdbc. Would you mind open a parent ticket and some subtasks, each one for one to-be-added dbms you've thought of?


On Sun, Feb 2, 2020 at 10:11 PM Jingsong Li <[hidden email]> wrote:
Yes, And I think we should add OracleDialect,SqlServerDialect,DB2Dialect support too.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 5:53 PM Flavio Pompermaier <[hidden email]> wrote:
Ok thanks for this info! Maybe this could be added to the documentation..what do you think?

Il Dom 2 Feb 2020, 08:37 Jingsong Li <[hidden email]> ha scritto:
Hi Flavio,

You can use `JDBCTableSource`, and register it from
 TableEnvionment.registerTableSource, you need provide
 a OracleDialect, maybe just implement `canHandle` and
 `defaultDriverName` is OK.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 2:42 PM Jark Wu <[hidden email]> wrote:
Hi Flavio,

If you want to adjust the writing statement for Oracle, you can implement the JDBCDialect for Oracle, and pass to the JDBCUpsertTableSink when constructing via `JDBCOptions.Builder#setDialect`. In this way, you don't need to recompile the source code of flink-jdbc.

Best,
Jark

On Fri, 31 Jan 2020 at 19:28, Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio


--
Best, Jingsong Lee


--
Best, Jingsong Lee


--
Best, Jingsong Lee


--
Flavio Pompermaier
Development Department

OKKAM S.r.l.
Tel. +(39) 0461 041809


--
Best, Jingsong Lee


--
Flavio Pompermaier
Development Department

OKKAM S.r.l.
Tel. +(39) 0461 041809
Reply | Threaded
Open this post in threaded view
|

Re: Read data from Oracle using Flink SQL API

Jingsong Li
 Hi Flavio,

We can just document after support dialect through JDBCTableFactory.

Best,
Jingsong Lee

On Thu, Feb 6, 2020 at 10:54 PM Flavio Pompermaier <[hidden email]> wrote:
Ok to me, I just want this to be documented in at https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/connect.html#jdbc-connector. Do I need to open another JIRA for that?

On Thu, Feb 6, 2020 at 11:04 AM Jingsong Li <[hidden email]> wrote:
Hi Flavio,

Instead of "TableEnvionment.registerTableSource", we want to drop it in 1.10.

I think you can create a JIRA to support pass dialect through JDBCTableFactory. We should support it. 

What do you think?

Best,
Jingsong Lee 

On Thu, Feb 6, 2020 at 6:01 PM Flavio Pompermaier <[hidden email]> wrote:
Should I open a JIRA to better document how to connect to an RDBMS that has no "out-of-the-box" dialect?
I.e. Pass the dialect impl in the TableEnvionment.registerTableSource?

On Thu, Feb 6, 2020 at 10:53 AM Jingsong Li <[hidden email]> wrote:
Hi Bowen,


Best,
Jingsong Lee

On Thu, Feb 6, 2020 at 12:57 PM Bowen Li <[hidden email]> wrote:
Hi Flavio,

+1 for adding Oracle (potentially more dbms like SqlServer, etc) to flink-jdbc. Would you mind open a parent ticket and some subtasks, each one for one to-be-added dbms you've thought of?


On Sun, Feb 2, 2020 at 10:11 PM Jingsong Li <[hidden email]> wrote:
Yes, And I think we should add OracleDialect,SqlServerDialect,DB2Dialect support too.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 5:53 PM Flavio Pompermaier <[hidden email]> wrote:
Ok thanks for this info! Maybe this could be added to the documentation..what do you think?

Il Dom 2 Feb 2020, 08:37 Jingsong Li <[hidden email]> ha scritto:
Hi Flavio,

You can use `JDBCTableSource`, and register it from
 TableEnvionment.registerTableSource, you need provide
 a OracleDialect, maybe just implement `canHandle` and
 `defaultDriverName` is OK.

Best,
Jingsong Lee

On Sun, Feb 2, 2020 at 2:42 PM Jark Wu <[hidden email]> wrote:
Hi Flavio,

If you want to adjust the writing statement for Oracle, you can implement the JDBCDialect for Oracle, and pass to the JDBCUpsertTableSink when constructing via `JDBCOptions.Builder#setDialect`. In this way, you don't need to recompile the source code of flink-jdbc.

Best,
Jark

On Fri, 31 Jan 2020 at 19:28, Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shouldn't have problems with dialects...am I wrong?
And what am I supposed to do right now if I want to connect to Oracle using the Table API? Do I have to use the low level JDBCInputFormat? Is there an easy way to connect to Oracle using the Table API without the need to modify and recompile the source code of Flink (just adding some interface implementation in the application JAR)?

Flavio


--
Best, Jingsong Lee


--
Best, Jingsong Lee


--
Best, Jingsong Lee


--
Flavio Pompermaier
Development Department

OKKAM S.r.l.
Tel. +(39) 0461 041809


--
Best, Jingsong Lee


--
Flavio Pompermaier
Development Department

OKKAM S.r.l.
Tel. +(39) 0461 041809


--
Best, Jingsong Lee