Is Queryable State not working in standalone cluster with 1 task manager?

[Oytun Tez]

Hi there,

We have a job that is run inside Docker via `standalone-job.sh start-foreground --job-classname`, with 1 task manager. I've been trying to make QueryableState available in this setup for 2 days now and I can't seem to enable it.

If I create a LocalEnvironment within the code itself and provide queryable-state.enable: true config directly via Configuration, then I can at least see that the queryable state server is spinning up.

Any pointers? With standalone-job.sh, it seems that it doesn't care about queryable-state.enable config inside flink-conf.yaml.

[Oytun Tez]

And, Flink version: 1.8, incl. Docker container (official flink:1.8 tag)

On Mon, Jul 29, 2019 at 4:32 PM Oytun Tez <[hidden email]> wrote:

Hi there,

We have a job that is run inside Docker via `standalone-job.sh start-foreground --job-classname`, with 1 task manager. I've been trying to make QueryableState available in this setup for 2 days now and I can't seem to enable it.

If I create a LocalEnvironment within the code itself and provide queryable-state.enable: true config directly via Configuration, then I can at least see that the queryable state server is spinning up.

Any pointers? With standalone-job.sh, it seems that it doesn't care about queryable-state.enable config inside flink-conf.yaml.

[Fabian Hueske-2]

Hi Oytun,

Is QS enabled in your Docker image or did you enable QS by copying/moving flink-queryable-state-runtime_2.11-1.8.0.jar from ./opt to ./lib [1]?

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#activating-queryable-state

Am Mo., 29. Juli 2019 um 22:46 Uhr schrieb Oytun Tez <[hidden email]>:

And, Flink version: 1.8, incl. Docker container (official flink:1.8 tag)

On Mon, Jul 29, 2019 at 4:32 PM Oytun Tez <[hidden email]> wrote:

Hi there,

We have a job that is run inside Docker via `standalone-job.sh start-foreground --job-classname`, with 1 task manager. I've been trying to make QueryableState available in this setup for 2 days now and I can't seem to enable it.

If I create a LocalEnvironment within the code itself and provide queryable-state.enable: true config directly via Configuration, then I can at least see that the queryable state server is spinning up.

Any pointers? With standalone-job.sh, it seems that it doesn't care about queryable-state.enable config inside flink-conf.yaml.

[Oytun Tez]

We extend the official container and the jar is available under ./lib – so it wasn't enabled by default, we moved the jar under ./lib and then build the container+application.

On Tue, Jul 30, 2019 at 3:42 AM Fabian Hueske <[hidden email]> wrote:

Hi Oytun,

Is QS enabled in your Docker image or did you enable QS by copying/moving flink-queryable-state-runtime_2.11-1.8.0.jar from ./opt to ./lib [1]?

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#activating-queryable-state

Am Mo., 29. Juli 2019 um 22:46 Uhr schrieb Oytun Tez <[hidden email]>:

And, Flink version: 1.8, incl. Docker container (official flink:1.8 tag)

On Mon, Jul 29, 2019 at 4:32 PM Oytun Tez <[hidden email]> wrote:

Hi there,

We have a job that is run inside Docker via `standalone-job.sh start-foreground --job-classname`, with 1 task manager. I've been trying to make QueryableState available in this setup for 2 days now and I can't seem to enable it.

If I create a LocalEnvironment within the code itself and provide queryable-state.enable: true config directly via Configuration, then I can at least see that the queryable state server is spinning up.

Any pointers? With standalone-job.sh, it seems that it doesn't care about queryable-state.enable config inside flink-conf.yaml.

[Oytun Tez]

On Tue, Jul 30, 2019 at 9:04 AM Oytun Tez <[hidden email]> wrote:

We extend the official container and the jar is available under ./lib – so it wasn't enabled by default, we moved the jar under ./lib and then build the container+application.

On Tue, Jul 30, 2019 at 3:42 AM Fabian Hueske <[hidden email]> wrote:

Hi Oytun,

Is QS enabled in your Docker image or did you enable QS by copying/moving flink-queryable-state-runtime_2.11-1.8.0.jar from ./opt to ./lib [1]?

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#activating-queryable-state

Am Mo., 29. Juli 2019 um 22:46 Uhr schrieb Oytun Tez <[hidden email]>:

And, Flink version: 1.8, incl. Docker container (official flink:1.8 tag)

On Mon, Jul 29, 2019 at 4:32 PM Oytun Tez <[hidden email]> wrote:

Hi there,

We have a job that is run inside Docker via `standalone-job.sh start-foreground --job-classname`, with 1 task manager. I've been trying to make QueryableState available in this setup for 2 days now and I can't seem to enable it.

If I create a LocalEnvironment within the code itself and provide queryable-state.enable: true config directly via Configuration, then I can at least see that the queryable state server is spinning up.

Any pointers? With standalone-job.sh, it seems that it doesn't care about queryable-state.enable config inside flink-conf.yaml.

[Oytun Tez]

On Tue, Jul 30, 2019 at 9:05 AM Oytun Tez <[hidden email]> wrote:

On Tue, Jul 30, 2019 at 9:04 AM Oytun Tez <[hidden email]> wrote:

We extend the official container and the jar is available under ./lib – so it wasn't enabled by default, we moved the jar under ./lib and then build the container+application.

On Tue, Jul 30, 2019 at 3:42 AM Fabian Hueske <[hidden email]> wrote:

Hi Oytun,

Is QS enabled in your Docker image or did you enable QS by copying/moving flink-queryable-state-runtime_2.11-1.8.0.jar from ./opt to ./lib [1]?

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#activating-queryable-state

Am Mo., 29. Juli 2019 um 22:46 Uhr schrieb Oytun Tez <[hidden email]>:

And, Flink version: 1.8, incl. Docker container (official flink:1.8 tag)

On Mon, Jul 29, 2019 at 4:32 PM Oytun Tez <[hidden email]> wrote:

Hi there,

We have a job that is run inside Docker via `standalone-job.sh start-foreground --job-classname`, with 1 task manager. I've been trying to make QueryableState available in this setup for 2 days now and I can't seem to enable it.

If I create a LocalEnvironment within the code itself and provide queryable-state.enable: true config directly via Configuration, then I can at least see that the queryable state server is spinning up.

Any pointers? With standalone-job.sh, it seems that it doesn't care about queryable-state.enable config inside flink-conf.yaml.

[Oytun Tez]

Both task manager and job manager are running in the same container, could that be the reason?

The container is using StandaloneJobClusterEntryPoint. Verified flink-conf and ./lib/runtime jar, both present, I can see the config is loaded from the logs, and nothing else about queryable state in the logs.

On Tue, Jul 30, 2019 at 9:05 AM Oytun Tez <[hidden email]> wrote:

On Tue, Jul 30, 2019 at 9:05 AM Oytun Tez <[hidden email]> wrote:

On Tue, Jul 30, 2019 at 9:04 AM Oytun Tez <[hidden email]> wrote:

We extend the official container and the jar is available under ./lib – so it wasn't enabled by default, we moved the jar under ./lib and then build the container+application.

On Tue, Jul 30, 2019 at 3:42 AM Fabian Hueske <[hidden email]> wrote:

Hi Oytun,

Is QS enabled in your Docker image or did you enable QS by copying/moving flink-queryable-state-runtime_2.11-1.8.0.jar from ./opt to ./lib [1]?

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#activating-queryable-state

Am Mo., 29. Juli 2019 um 22:46 Uhr schrieb Oytun Tez <[hidden email]>:

And, Flink version: 1.8, incl. Docker container (official flink:1.8 tag)

On Mon, Jul 29, 2019 at 4:32 PM Oytun Tez <[hidden email]> wrote:

Hi there,

We have a job that is run inside Docker via `standalone-job.sh start-foreground --job-classname`, with 1 task manager. I've been trying to make QueryableState available in this setup for 2 days now and I can't seem to enable it.

If I create a LocalEnvironment within the code itself and provide queryable-state.enable: true config directly via Configuration, then I can at least see that the queryable state server is spinning up.

Any pointers? With standalone-job.sh, it seems that it doesn't care about queryable-state.enable config inside flink-conf.yaml.

[Fabian Hueske-2]

JM, TM and QS client are communicating via the network.

I'm not super familiar with Docker setups, but you might need to open some ports to make it work.

I'd check the default port settings for the QS configuration and open them.

Am Di., 30. Juli 2019 um 16:11 Uhr schrieb Oytun Tez <[hidden email]>:

Both task manager and job manager are running in the same container, could that be the reason?

The container is using StandaloneJobClusterEntryPoint. Verified flink-conf and ./lib/runtime jar, both present, I can see the config is loaded from the logs, and nothing else about queryable state in the logs.

On Tue, Jul 30, 2019 at 9:05 AM Oytun Tez <[hidden email]> wrote:

On Tue, Jul 30, 2019 at 9:05 AM Oytun Tez <[hidden email]> wrote:

On Tue, Jul 30, 2019 at 9:04 AM Oytun Tez <[hidden email]> wrote:

We extend the official container and the jar is available under ./lib – so it wasn't enabled by default, we moved the jar under ./lib and then build the container+application.

On Tue, Jul 30, 2019 at 3:42 AM Fabian Hueske <[hidden email]> wrote:

Hi Oytun,

Is QS enabled in your Docker image or did you enable QS by copying/moving flink-queryable-state-runtime_2.11-1.8.0.jar from ./opt to ./lib [1]?

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#activating-queryable-state

Am Mo., 29. Juli 2019 um 22:46 Uhr schrieb Oytun Tez <[hidden email]>:

And, Flink version: 1.8, incl. Docker container (official flink:1.8 tag)

On Mon, Jul 29, 2019 at 4:32 PM Oytun Tez <[hidden email]> wrote:

Hi there,

We have a job that is run inside Docker via `standalone-job.sh start-foreground --job-classname`, with 1 task manager. I've been trying to make QueryableState available in this setup for 2 days now and I can't seem to enable it.

If I create a LocalEnvironment within the code itself and provide queryable-state.enable: true config directly via Configuration, then I can at least see that the queryable state server is spinning up.

Any pointers? With standalone-job.sh, it seems that it doesn't care about queryable-state.enable config inside flink-conf.yaml.

[Oytun Tez]

Wow, I think QS state and proxy servers are actually running, but the logs are not sent to container logs.

I was able to see the activation in /opt/flink/log/taskexecutor-*.log file:

These lines however are not visible in the typical docker container log.

Thank you for the support, Fabian. I am now structuring our QS interactions, let's see what comes.

On Tue, Jul 30, 2019 at 10:15 AM Fabian Hueske <[hidden email]> wrote:

JM, TM and QS client are communicating via the network.

I'm not super familiar with Docker setups, but you might need to open some ports to make it work.

I'd check the default port settings for the QS configuration and open them.

Am Di., 30. Juli 2019 um 16:11 Uhr schrieb Oytun Tez <[hidden email]>:

Both task manager and job manager are running in the same container, could that be the reason?

The container is using StandaloneJobClusterEntryPoint. Verified flink-conf and ./lib/runtime jar, both present, I can see the config is loaded from the logs, and nothing else about queryable state in the logs.

On Tue, Jul 30, 2019 at 9:05 AM Oytun Tez <[hidden email]> wrote:

On Tue, Jul 30, 2019 at 9:05 AM Oytun Tez <[hidden email]> wrote:

On Tue, Jul 30, 2019 at 9:04 AM Oytun Tez <[hidden email]> wrote:

We extend the official container and the jar is available under ./lib – so it wasn't enabled by default, we moved the jar under ./lib and then build the container+application.

On Tue, Jul 30, 2019 at 3:42 AM Fabian Hueske <[hidden email]> wrote:

Hi Oytun,

Is QS enabled in your Docker image or did you enable QS by copying/moving flink-queryable-state-runtime_2.11-1.8.0.jar from ./opt to ./lib [1]?

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#activating-queryable-state

Am Mo., 29. Juli 2019 um 22:46 Uhr schrieb Oytun Tez <[hidden email]>:

And, Flink version: 1.8, incl. Docker container (official flink:1.8 tag)

On Mon, Jul 29, 2019 at 4:32 PM Oytun Tez <[hidden email]> wrote:

Hi there,

We have a job that is run inside Docker via `standalone-job.sh start-foreground --job-classname`, with 1 task manager. I've been trying to make QueryableState available in this setup for 2 days now and I can't seem to enable it.

If I create a LocalEnvironment within the code itself and provide queryable-state.enable: true config directly via Configuration, then I can at least see that the queryable state server is spinning up.

Any pointers? With standalone-job.sh, it seems that it doesn't care about queryable-state.enable config inside flink-conf.yaml.