Does Flink has a JDBC server where I can submit Calcite Streaming Queries?

[kant kodali]

Hi All,

Does Flink has a JDBC server where I can submit Calcite Streaming Queries? such that I get Stream of responses back from Flink forever via JDBC ? What is the standard way to do this?

Thanks,

Kant

[Fabian Hueske-2]

Hi Kant,

no, there is no such functionality.

I'm also not sure how well streaming would work together with the JDBC interface. JDBC has not been designed for continuous streaming queries, i.e., queries that never terminate.

Challenges would be to have an infinite, streamable ResultSet (which might be possible) and how to represent retractions, i.e., updates of previously emitted results (I doubt this would work).

If a streamable ResultSet was possible, a subset of queries (those that only produce new rows and never have to update emitted results) could be supported.

Right now, the approach would be to implement a client program that executes queries and writes their result to a destination like Kafka or a database using a TableSink [1].

The community is also discussing a SQL client to submits queries. [2]

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table/sourceSinks.html
[2] https://issues.apache.org/jira/browse/FLINK-7594

2017-09-07 21:43 GMT+02:00 kant kodali <[hidden email]>:

Hi All,

Does Flink has a JDBC server where I can submit Calcite Streaming Queries? such that I get Stream of responses back from Flink forever via JDBC ? What is the standard way to do this?

Thanks,

Kant

[kant kodali]

Hi Fabian,

Thanks for the response. I understand the common approach is to write a client program and run it however this will not allow me to send queries Ad-hoc so Is there anyway for me to submit Calcite SQL to Flink via REST or whatever mechanism? Forgot even, the result set once I know there is a way to submit Adhoc queries I can figure out a way to write to Kafka.

Thanks,

kant

On Fri, Sep 8, 2017 at 12:52 AM, Fabian Hueske <[hidden email]> wrote:

Hi Kant,

no, there is no such functionality.

I'm also not sure how well streaming would work together with the JDBC interface. JDBC has not been designed for continuous streaming queries, i.e., queries that never terminate.

Challenges would be to have an infinite, streamable ResultSet (which might be possible) and how to represent retractions, i.e., updates of previously emitted results (I doubt this would work).

If a streamable ResultSet was possible, a subset of queries (those that only produce new rows and never have to update emitted results) could be supported.

Right now, the approach would be to implement a client program that executes queries and writes their result to a destination like Kafka or a database using a TableSink [1].

The community is also discussing a SQL client to submits queries. [2]

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table/sourceSinks.html
[2] https://issues.apache.org/jira/browse/FLINK-7594

2017-09-07 21:43 GMT+02:00 kant kodali <[hidden email]>:

Hi All,

Does Flink has a JDBC server where I can submit Calcite Streaming Queries? such that I get Stream of responses back from Flink forever via JDBC ? What is the standard way to do this?

Thanks,

Kant

[Fabian Hueske-2]

As I said, there is no such functionality built into Flink yet.

A client program can be parameterized with a query and turned into a SQL client that way.

The submission would work with the regular Flink job client, i.e., it would pickup the regular Flink config.

Best, Fabian

2017-09-08 10:05 GMT+02:00 kant kodali <[hidden email]>:

Hi Fabian,

Thanks for the response. I understand the common approach is to write a client program and run it however this will not allow me to send queries Ad-hoc so Is there anyway for me to submit Calcite SQL to Flink via REST or whatever mechanism? Forgot even, the result set once I know there is a way to submit Adhoc queries I can figure out a way to write to Kafka.

Thanks,

kant

On Fri, Sep 8, 2017 at 12:52 AM, Fabian Hueske <[hidden email]> wrote:

Hi Kant,

no, there is no such functionality.

I'm also not sure how well streaming would work together with the JDBC interface. JDBC has not been designed for continuous streaming queries, i.e., queries that never terminate.

Challenges would be to have an infinite, streamable ResultSet (which might be possible) and how to represent retractions, i.e., updates of previously emitted results (I doubt this would work).

If a streamable ResultSet was possible, a subset of queries (those that only produce new rows and never have to update emitted results) could be supported.

Right now, the approach would be to implement a client program that executes queries and writes their result to a destination like Kafka or a database using a TableSink [1].

The community is also discussing a SQL client to submits queries. [2]

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table/sourceSinks.html
[2] https://issues.apache.org/jira/browse/FLINK-7594

2017-09-07 21:43 GMT+02:00 kant kodali <[hidden email]>:

Hi All,

Does Flink has a JDBC server where I can submit Calcite Streaming Queries? such that I get Stream of responses back from Flink forever via JDBC ? What is the standard way to do this?

Thanks,

Kant

[kant kodali]

Got it! Thanks a lot for your detailed explanation. 

On Fri, Sep 8, 2017 at 1:27 AM, Fabian Hueske <[hidden email]> wrote:

As I said, there is no such functionality built into Flink yet.

A client program can be parameterized with a query and turned into a SQL client that way.

The submission would work with the regular Flink job client, i.e., it would pickup the regular Flink config.

Best, Fabian

2017-09-08 10:05 GMT+02:00 kant kodali <[hidden email]>:

Hi Fabian,

Thanks for the response. I understand the common approach is to write a client program and run it however this will not allow me to send queries Ad-hoc so Is there anyway for me to submit Calcite SQL to Flink via REST or whatever mechanism? Forgot even, the result set once I know there is a way to submit Adhoc queries I can figure out a way to write to Kafka.

Thanks,

kant

On Fri, Sep 8, 2017 at 12:52 AM, Fabian Hueske <[hidden email]> wrote:

Hi Kant,

no, there is no such functionality.

I'm also not sure how well streaming would work together with the JDBC interface. JDBC has not been designed for continuous streaming queries, i.e., queries that never terminate.

Challenges would be to have an infinite, streamable ResultSet (which might be possible) and how to represent retractions, i.e., updates of previously emitted results (I doubt this would work).

If a streamable ResultSet was possible, a subset of queries (those that only produce new rows and never have to update emitted results) could be supported.

Right now, the approach would be to implement a client program that executes queries and writes their result to a destination like Kafka or a database using a TableSink [1].

The community is also discussing a SQL client to submits queries. [2]

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table/sourceSinks.html
[2] https://issues.apache.org/jira/browse/FLINK-7594

2017-09-07 21:43 GMT+02:00 kant kodali <[hidden email]>:

Hi All,

Does Flink has a JDBC server where I can submit Calcite Streaming Queries? such that I get Stream of responses back from Flink forever via JDBC ? What is the standard way to do this?

Thanks,

Kant