Flink SQL client support for running in Flink cluster

[Dipanjan Mazumder]

Hi Guys,

    I was going through the Flink sql client configuration YAML from the training example and came across a section in the configuration as below:

############################################################     

# Deployment properties allow for describing the cluster to which table

# programs are submitted to.

deployment:

  type: standalone             # only the 'standalone' deployment is supported

  response-timeout: 5000       # general cluster communication timeout in ms

  gateway-address: ""          # (optional) address from cluster to gateway

  gateway-port: 0              # (optional) port from cluster to gateway

##############################################################

The first property of the deployment section "type" says that only standalone deployment of the program is supported , does that mean that Flink SQL client is non functional in when Flink running in Cluster mode. Also i cannot find any documentation for this configuration stating any different value of "deployment.type" for example like "cluster" or something. 

I want to implement an API on the SQL client which will support dynamic submission of configurations and thereby sql queries on the fly for the new configuration. If Flink sql is non-functional in Flink cluster then the effort will be not worth because in that case i will have to first see how the program can be made to work in Flink cluster first and then implement the API on top of that.

Please kindly help me on the same , as i am stuck for further changes and i am still not going in to identify if what i see is true because it will take more time to identify through code which can be probably answered faster by your team.

Regards

Dipanjan

[Xingcan Cui]

Hi Dipanjan,

Here the ’standalone’ is short for ’standalone cluster mode’, which means you can setup a standalone Flink cluster and submit your SQL queries (via SQL client) to the cluster.

As far as I know, the current SQL client can only connect to a Flink cluster deployed in standalone mode.

Hope that helps.

Best,

Xingcan

On Sep 7, 2019, at 1:57 AM, Dipanjan Mazumder <[hidden email]> wrote:

Hi Guys,

    I was going through the Flink sql client configuration YAML from the training example and came across a section in the configuration as below:

############################################################     

# Deployment properties allow for describing the cluster to which table

# programs are submitted to.

deployment:

  type: standalone             # only the 'standalone' deployment is supported

  response-timeout: 5000       # general cluster communication timeout in ms

  gateway-address: ""          # (optional) address from cluster to gateway

  gateway-port: 0              # (optional) port from cluster to gateway

##############################################################

The first property of the deployment section "type" says that only standalone deployment of the program is supported , does that mean that Flink SQL client is non functional in when Flink running in Cluster mode. Also i cannot find any documentation for this configuration stating any different value of "deployment.type" for example like "cluster" or something. 

I want to implement an API on the SQL client which will support dynamic submission of configurations and thereby sql queries on the fly for the new configuration. If Flink sql is non-functional in Flink cluster then the effort will be not worth because in that case i will have to first see how the program can be made to work in Flink cluster first and then implement the API on top of that.

Please kindly help me on the same , as i am stuck for further changes and i am still not going in to identify if what i see is true because it will take more time to identify through code which can be probably answered faster by your team.

Regards

Dipanjan

[Dipanjan Mazumder]

Hi Xingcan,

    Thanks a lot for this info and this verifies my suspect , so if i want to reuse the SQL client to implement an API based framework , it will not work in Flink cluster running in  cluster mode, is there any plan in near future to support that, if its planned is there any expected release event for the same ,because i need to prototype my platform and present the same.

Or is there any other way i can achieve this dynamic configuration and query submission to Flink running as a full fledged cluster.

Regards

Dipanjan

On Saturday, September 7, 2019, 12:52:02 PM GMT+5:30, Xingcan Cui <[hidden email]> wrote:

Hi Dipanjan,

Here the ’standalone’ is short for ’standalone cluster mode’, which means you can setup a standalone Flink cluster and submit your SQL queries (via SQL client) to the cluster.

As far as I know, the current SQL client can only connect to a Flink cluster deployed in standalone mode.

Hope that helps.

Best,

Xingcan

On Sep 7, 2019, at 1:57 AM, Dipanjan Mazumder <[hidden email]> wrote:

Hi Guys,

    I was going through the Flink sql client configuration YAML from the training example and came across a section in the configuration as below:

############################################################     

# Deployment properties allow for describing the cluster to which table

# programs are submitted to.

deployment:

  type: standalone             # only the 'standalone' deployment is supported

  response-timeout: 5000       # general cluster communication timeout in ms

  gateway-address: ""          # (optional) address from cluster to gateway

  gateway-port: 0              # (optional) port from cluster to gateway

##############################################################

The first property of the deployment section "type" says that only standalone deployment of the program is supported , does that mean that Flink SQL client is non functional in when Flink running in Cluster mode. Also i cannot find any documentation for this configuration stating any different value of "deployment.type" for example like "cluster" or something. 

I want to implement an API on the SQL client which will support dynamic submission of configurations and thereby sql queries on the fly for the new configuration. If Flink sql is non-functional in Flink cluster then the effort will be not worth because in that case i will have to first see how the program can be made to work in Flink cluster first and then implement the API on top of that.

Please kindly help me on the same , as i am stuck for further changes and i am still not going in to identify if what i see is true because it will take more time to identify through code which can be probably answered faster by your team.

Regards

Dipanjan

[Xingcan Cui]

Hi Dipanjan,

Sorry that I didn’t make it clear. The ’standalone cluster’ could be fully distributed. It refers to a static cluster deployed without any cluster managers (or other related tools), such as YARN and Kubernetes.

For more information, please check the documentation [1].

Best,

Xingcan

[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/cluster_setup.html

On Sep 7, 2019, at 3:30 AM, Dipanjan Mazumder <[hidden email]> wrote:

Hi Xingcan,

    Thanks a lot for this info and this verifies my suspect , so if i want to reuse the SQL client to implement an API based framework , it will not work in Flink cluster running in  cluster mode, is there any plan in near future to support that, if its planned is there any expected release event for the same ,because i need to prototype my platform and present the same.

Or is there any other way i can achieve this dynamic configuration and query submission to Flink running as a full fledged cluster.

Regards

Dipanjan

On Saturday, September 7, 2019, 12:52:02 PM GMT+5:30, Xingcan Cui <[hidden email]> wrote:

Hi Dipanjan,

Here the ’standalone’ is short for ’standalone cluster mode’, which means you can setup a standalone Flink cluster and submit your SQL queries (via SQL client) to the cluster.

As far as I know, the current SQL client can only connect to a Flink cluster deployed in standalone mode.

Hope that helps.

Best,

Xingcan

On Sep 7, 2019, at 1:57 AM, Dipanjan Mazumder <[hidden email]> wrote:

Hi Guys,

    I was going through the Flink sql client configuration YAML from the training example and came across a section in the configuration as below:

############################################################     

# Deployment properties allow for describing the cluster to which table

# programs are submitted to.

deployment:

  type: standalone             # only the 'standalone' deployment is supported

  response-timeout: 5000       # general cluster communication timeout in ms

  gateway-address: ""          # (optional) address from cluster to gateway

  gateway-port: 0              # (optional) port from cluster to gateway

##############################################################

The first property of the deployment section "type" says that only standalone deployment of the program is supported , does that mean that Flink SQL client is non functional in when Flink running in Cluster mode. Also i cannot find any documentation for this configuration stating any different value of "deployment.type" for example like "cluster" or something. 

I want to implement an API on the SQL client which will support dynamic submission of configurations and thereby sql queries on the fly for the new configuration. If Flink sql is non-functional in Flink cluster then the effort will be not worth because in that case i will have to first see how the program can be made to work in Flink cluster first and then implement the API on top of that.

Please kindly help me on the same , as i am stuck for further changes and i am still not going in to identify if what i see is true because it will take more time to identify through code which can be probably answered faster by your team.

Regards

Dipanjan