Queryable State in Flink 1.4

[Boris Lublinsky]

It appears, that queryable state access significantly changed in 1.4 compared to 1.3.

Documentation on the queryable state client https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html#example

States that the client needs to connect to a proxy port.

My implementation, which I used for 1.3 is enclosed 

Here I am setting both server port and proxy port.

When I am running it on a localhost and try to do lsof -i:9069 and lsof -i:9067 It does not show anything using this port.

Am I missing something?

As a result my query implementation 

  

Return an error - connection refused

My state is defined in the following class 

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Till Rohrmann]

Hi Boris,

if you start 2 TaskManagers on the same host, then you have to define a port range for the KvState server and the proxy. Otherwise the Flink cluster should not be able to start.

Cheers,

Till

On Thu, Jan 4, 2018 at 11:19 PM, Boris Lublinsky <[hidden email]> wrote:

It appears, that queryable state access significantly changed in 1.4 compared to 1.3.

Documentation on the queryable state client https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html#example

States that the client needs to connect to a proxy port.

My implementation, which I used for 1.3 is enclosed 

Here I am setting both server port and proxy port.

When I am running it on a localhost and try to do lsof -i:9069 and lsof -i:9067 It does not show anything using this port.

Am I missing something?

As a result my query implementation 

  

Return an error - connection refused

My state is defined in the following class 

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Boris Lublinsky]

Thanks Till,

I am probably slow.

I changed the code to the following:

// In a non MiniCluster setup queryable state is enabled by default.
config.setString(QueryableStateOptions.PROXY_PORT_RANGE, "50100-50101")
config.setInteger(QueryableStateOptions.PROXY_NETWORK_THREADS, 2)
config.setInteger(QueryableStateOptions.PROXY_ASYNC_QUERY_THREADS, 2)

config.setString(QueryableStateOptions.SERVER_PORT_RANGE, "50110-50111")
config.setInteger(QueryableStateOptions.SERVER_NETWORK_THREADS, 2)
config.setInteger(QueryableStateOptions.SERVER_ASYNC_QUERY_THREADS, 2)

And lsof still shows nothing 

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Jan 5, 2018, at 6:33 AM, Till Rohrmann <[hidden email]> wrote:

Hi Boris,

if you start 2 TaskManagers on the same host, then you have to define a port range for the KvState server and the proxy. Otherwise the Flink cluster should not be able to start.

Cheers,

Till

On Thu, Jan 4, 2018 at 11:19 PM, Boris Lublinsky <[hidden email]> wrote:

It appears, that queryable state access significantly changed in 1.4 compared to 1.3.

Documentation on the queryable state client https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html#example

States that the client needs to connect to a proxy port.

My implementation, which I used for 1.3 is enclosed 

Here I am setting both server port and proxy port.

When I am running it on a localhost and try to do lsof -i:9069 and lsof -i:9067 It does not show anything using this port.

Am I missing something?

As a result my query implementation 

  

Return an error - connection refused

My state is defined in the following class 

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Boris Lublinsky]

I also tried to comment out 

//    config.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 2);

Still no luck.

Do you guys have a working example for queryable state for 1.4 somewhere?

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Jan 5, 2018, at 6:33 AM, Till Rohrmann <[hidden email]> wrote:

Hi Boris,

if you start 2 TaskManagers on the same host, then you have to define a port range for the KvState server and the proxy. Otherwise the Flink cluster should not be able to start.

Cheers,

Till

On Thu, Jan 4, 2018 at 11:19 PM, Boris Lublinsky <[hidden email]> wrote:

It appears, that queryable state access significantly changed in 1.4 compared to 1.3.

Documentation on the queryable state client https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html#example

States that the client needs to connect to a proxy port.

My implementation, which I used for 1.3 is enclosed 

Here I am setting both server port and proxy port.

When I am running it on a localhost and try to do lsof -i:9069 and lsof -i:9067 It does not show anything using this port.

Am I missing something?

As a result my query implementation 

  

Return an error - connection refused

My state is defined in the following class 

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Till Rohrmann]

Did you add the `flink-queryable-state-runtime` jar as a dependency to your project? You can check the log whether a queryable state proxy and server have been started.

Cheers,

Till

On Fri, Jan 5, 2018 at 5:19 PM, Boris Lublinsky <[hidden email]> wrote:

I also tried to comment out 

//    config.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 2);

Still no luck.

Do you guys have a working example for queryable state for 1.4 somewhere?

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Jan 5, 2018, at 6:33 AM, Till Rohrmann <[hidden email]> wrote:

Hi Boris,

if you start 2 TaskManagers on the same host, then you have to define a port range for the KvState server and the proxy. Otherwise the Flink cluster should not be able to start.

Cheers,

Till

On Thu, Jan 4, 2018 at 11:19 PM, Boris Lublinsky <[hidden email]> wrote:

It appears, that queryable state access significantly changed in 1.4 compared to 1.3.

Documentation on the queryable state client https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html#example

States that the client needs to connect to a proxy port.

My implementation, which I used for 1.3 is enclosed 

Here I am setting both server port and proxy port.

When I am running it on a localhost and try to do lsof -i:9069 and lsof -i:9067 It does not show anything using this port.

Am I missing something?

As a result my query implementation 

  

Return an error - connection refused

My state is defined in the following class 

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Boris Lublinsky]

Thanks 

This was it.

It would help to have this in documentation along with `flink-queryable-state-client`

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Jan 5, 2018, at 11:46 AM, Till Rohrmann <[hidden email]> wrote:

Did you add the `flink-queryable-state-runtime` jar as a dependency to your project? You can check the log whether a queryable state proxy and server have been started.

Cheers,

Till

On Fri, Jan 5, 2018 at 5:19 PM, Boris Lublinsky <[hidden email]> wrote:

I also tried to comment out 

//    config.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 2);

Still no luck.

Do you guys have a working example for queryable state for 1.4 somewhere?

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Jan 5, 2018, at 6:33 AM, Till Rohrmann <[hidden email]> wrote:

Hi Boris,

if you start 2 TaskManagers on the same host, then you have to define a port range for the KvState server and the proxy. Otherwise the Flink cluster should not be able to start.

Cheers,

Till

On Thu, Jan 4, 2018 at 11:19 PM, Boris Lublinsky <[hidden email]> wrote:

It appears, that queryable state access significantly changed in 1.4 compared to 1.3.

Documentation on the queryable state client https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html#example

States that the client needs to connect to a proxy port.

My implementation, which I used for 1.3 is enclosed 

Here I am setting both server port and proxy port.

When I am running it on a localhost and try to do lsof -i:9069 and lsof -i:9067 It does not show anything using this port.

Am I missing something?

As a result my query implementation 

  

Return an error - connection refused

My state is defined in the following class 

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Fabian Hueske-2]

Thanks Boris.

I've filed FLINK-8391 [1] to extend the documentation.

Best, Fabian

[1] https://issues.apache.org/jira/browse/FLINK-8391

2018-01-05 19:52 GMT+01:00 Boris Lublinsky <[hidden email]>:

Thanks 

This was it.

It would help to have this in documentation along with `flink-queryable-state-client`

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Jan 5, 2018, at 11:46 AM, Till Rohrmann <[hidden email]> wrote:

Did you add the `flink-queryable-state-runtime` jar as a dependency to your project? You can check the log whether a queryable state proxy and server have been started.

Cheers,

Till

On Fri, Jan 5, 2018 at 5:19 PM, Boris Lublinsky <[hidden email]> wrote:

I also tried to comment out 

//    config.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 2);

Still no luck.

Do you guys have a working example for queryable state for 1.4 somewhere?

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Jan 5, 2018, at 6:33 AM, Till Rohrmann <[hidden email]> wrote:

Hi Boris,

if you start 2 TaskManagers on the same host, then you have to define a port range for the KvState server and the proxy. Otherwise the Flink cluster should not be able to start.

Cheers,

Till

On Thu, Jan 4, 2018 at 11:19 PM, Boris Lublinsky <[hidden email]> wrote:

It appears, that queryable state access significantly changed in 1.4 compared to 1.3.

Documentation on the queryable state client https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html#example

States that the client needs to connect to a proxy port.

My implementation, which I used for 1.3 is enclosed 

Here I am setting both server port and proxy port.

When I am running it on a localhost and try to do lsof -i:9069 and lsof -i:9067 It does not show anything using this port.

Am I missing something?

As a result my query implementation 

  

Return an error - connection refused

My state is defined in the following class 

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/