Extending Flink's SQL-Parser

Hey there,

 

I have to extend Flink's SQL-parser such that it accepts and evaluates select-queries with different syntax.

Furthermore I use Eclipse Oxygen with Maven plugin and Flink Release 1.8.0.

 

What I believe to know:

For parsing SQL-queries Flink-Table uses Apache Calcite's SQL-parser. Flink-Table-Planner is the only module that references the Calcite-Core which contains the parser ("Parser.jj" ?).

Therefore I want to import Flink-Table-Planner and Calcite-Core as local projects in Eclipse and edit the files "config.fmpp" and "parserImpls.ftl". After that I want to create a new "Parser.jj" file with Apache Freemaker (I assume there are some tutorials?).

 

What I don't know:

Is it a promising plan or are there better strategies to extend the parser?

I already tried to import Flink-Table-Planner but I got many errors which might refer to a Scala-problem with Eclipse. Do I have to switch to say IntelliJ? Furthermore I'm not yet clear about how exactly I can extend the parser. Are there any manuals/tutorials to teach me adding a new SELECT-syntax? I already came across the parser extension test but it didn't give me the answers I was looking for.

 

Thanks for your help!

 

Regards,

Dominik Gröninger

 



[Rui Li]

Hi Dominik,

I think you can check "parserImpls.ftl" to find out how Flink extends Calcite's original syntax to support features like CREATE TABLE and DROP TABLE, and follow those examples to implement your own syntax. It may also be helpful to check the pom.xml of flink-sql-parser to see how we use javacc plugin to generate the parser code.

At the moment I don't think there's any tutorials about extending the SQL parser because it's quite internal to Flink. But perhaps the following answer provides some insights about how to extend Calcite parser in general: https://stackoverflow.com/questions/44382826/how-to-change-calcites-default-sql-grammar

On Tue, Sep 17, 2019 at 12:16 AM [hidden email] <[hidden email]> wrote:

Hey there,

 

I have to extend Flink's SQL-parser such that it accepts and evaluates select-queries with different syntax.

Furthermore I use Eclipse Oxygen with Maven plugin and Flink Release 1.8.0.

 

What I believe to know:

For parsing SQL-queries Flink-Table uses Apache Calcite's SQL-parser. Flink-Table-Planner is the only module that references the Calcite-Core which contains the parser ("Parser.jj" ?).

Therefore I want to import Flink-Table-Planner and Calcite-Core as local projects in Eclipse and edit the files "config.fmpp" and "parserImpls.ftl". After that I want to create a new "Parser.jj" file with Apache Freemaker (I assume there are some tutorials?).

 

What I don't know:

Is it a promising plan or are there better strategies to extend the parser?

I already tried to import Flink-Table-Planner but I got many errors which might refer to a Scala-problem with Eclipse. Do I have to switch to say IntelliJ? Furthermore I'm not yet clear about how exactly I can extend the parser. Are there any manuals/tutorials to teach me adding a new SELECT-syntax? I already came across the parser extension test but it didn't give me the answers I was looking for.

 

Thanks for your help!

 

Regards,

Dominik Gröninger

 



--

[Rong Rong]

Hi Dominik,

To add to Rui's answer. there are other examples I can think of on how to extend Calcite's DDL syntax is already in Calcite's Server module [1] and one of our open-sourced project [2]. you might want to check them out. 

--

Rong

[1] https://github.com/apache/calcite/blob/master/server/src/main/codegen/includes/parserImpls.ftl

[2] https://github.com/uber/AthenaX/blob/master/athenax-vm-compiler/src/codegen/includes/parserImpls.ftl

On Mon, Sep 16, 2019 at 8:28 PM Rui Li <[hidden email]> wrote:

Hi Dominik,

I think you can check "parserImpls.ftl" to find out how Flink extends Calcite's original syntax to support features like CREATE TABLE and DROP TABLE, and follow those examples to implement your own syntax. It may also be helpful to check the pom.xml of flink-sql-parser to see how we use javacc plugin to generate the parser code.

At the moment I don't think there's any tutorials about extending the SQL parser because it's quite internal to Flink. But perhaps the following answer provides some insights about how to extend Calcite parser in general: https://stackoverflow.com/questions/44382826/how-to-change-calcites-default-sql-grammar

On Tue, Sep 17, 2019 at 12:16 AM [hidden email] <[hidden email]> wrote:

Hey there,

 

I have to extend Flink's SQL-parser such that it accepts and evaluates select-queries with different syntax.

Furthermore I use Eclipse Oxygen with Maven plugin and Flink Release 1.8.0.

 

What I believe to know:

For parsing SQL-queries Flink-Table uses Apache Calcite's SQL-parser. Flink-Table-Planner is the only module that references the Calcite-Core which contains the parser ("Parser.jj" ?).

Therefore I want to import Flink-Table-Planner and Calcite-Core as local projects in Eclipse and edit the files "config.fmpp" and "parserImpls.ftl". After that I want to create a new "Parser.jj" file with Apache Freemaker (I assume there are some tutorials?).

 

What I don't know:

Is it a promising plan or are there better strategies to extend the parser?

I already tried to import Flink-Table-Planner but I got many errors which might refer to a Scala-problem with Eclipse. Do I have to switch to say IntelliJ? Furthermore I'm not yet clear about how exactly I can extend the parser. Are there any manuals/tutorials to teach me adding a new SELECT-syntax? I already came across the parser extension test but it didn't give me the answers I was looking for.

 

Thanks for your help!

 

Regards,

Dominik Gröninger

 



--