(DEPRECATED) Apache Flink User Mailing List archive.

Unable to change job manager port when launching session cluster on Docker

Classic

List

Threaded

5 messages Options

Papadopoulos, Konstantinos

Unable to change job manager port when launching session cluster on Docker

Hello all,

I am trying to launch an Apache Flink session cluster on Docker using Docker Compose and following the respective tutorial:

https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/docker.html#flink-with-docker-compose

The default job manager port (i.e., 8081) is in use on my host so the cluster fails to launch.

I tried to change the configuration from the respective service definition on docker-compose.yml (e.g., services -> jobmanager -> ports: "5000:5000") with no success; job manager container seems to launch on the default port.

Can anybody help me to proceed with this?

Thanks in advance,

Konstantinos

P.S.: My docker-compose.yml is the following:

version: "2.1"

services:

jobmanager:

image: ${FLINK_DOCKER_IMAGE_NAME:-flink}

expose:

- "6123"

ports:

- "5000:5000"

command: jobmanager

environment:

- JOB_MANAGER_RPC_ADDRESS=jobmanager

taskmanager:

image: ${FLINK_DOCKER_IMAGE_NAME:-flink}

expose:

- "6121"

- "6122"

depends_on:

- jobmanager

command: taskmanager

links:

- "jobmanager:jobmanager"

environment:

- JOB_MANAGER_RPC_ADDRESS=jobmanager

Aleksey Pak

Re: Unable to change job manager port when launching session cluster on Docker

Hi Konstantinos,

Can you try using "8081:5000" as port binding configuration?

This should bind container's 8081 port to 5000 port on host.

If you want to use 5000 port as JobManager's port in the *container*, you would need to change flink-conf.yaml or pass additional command line argument (to override the corresponding config option).

Regards,

Aleksey

On Fri, Oct 18, 2019 at 2:55 PM Papadopoulos, Konstantinos <[hidden email]> wrote:

Hello all,

I am trying to launch an Apache Flink session cluster on Docker using Docker Compose and following the respective tutorial:

https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/docker.html#flink-with-docker-compose

The default job manager port (i.e., 8081) is in use on my host so the cluster fails to launch.

I tried to change the configuration from the respective service definition on docker-compose.yml (e.g., services -> jobmanager -> ports: "5000:5000") with no success; job manager container seems to launch on the default port.

Can anybody help me to proceed with this?

Thanks in advance,

Konstantinos

P.S.: My docker-compose.yml is the following:

version: "2.1"

services:

jobmanager:

    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}

    expose:

      - "6123"

    ports:

      - "5000:5000"

    command: jobmanager

    environment:

      - JOB_MANAGER_RPC_ADDRESS=jobmanager

taskmanager:

    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}

    expose:

      - "6121"

      - "6122"

    depends_on:

      - jobmanager

    command: taskmanager

    links:

      - "jobmanager:jobmanager"

    environment:

      - JOB_MANAGER_RPC_ADDRESS=jobmanager

Papadopoulos, Konstantinos

RE: Unable to change job manager port when launching session cluster on Docker

Hi Aleksey,

I tried using "8081:5000" as port binding configuration with no success. I also tried different port numbers (i.e,. other than 5000) to bind, but admin seems not to launch.

Is there any easy way to change flink-conf.yaml or pass additional command line argument keeping the docker-compose approach? Or alternatively, should I pull the image, create the container and modify the respective configuration after connecting to it?

Regards,

Konstantinos

From: Aleksey Pak <[hidden email]>
Sent: Friday, October 18, 2019 10:46 PM
To: Papadopoulos, Konstantinos <[hidden email]>
Cc: [hidden email]
Subject: Re: Unable to change job manager port when launching session cluster on Docker

Hi Konstantinos,

Can you try using "8081:5000" as port binding configuration?

This should bind container's 8081 port to 5000 port on host.

If you want to use 5000 port as JobManager's port in the *container*, you would need to change flink-conf.yaml or pass additional command line argument (to override the corresponding config option).

Regards,

Aleksey

On Fri, Oct 18, 2019 at 2:55 PM Papadopoulos, Konstantinos <[hidden email]> wrote:

Hello all,

I am trying to launch an Apache Flink session cluster on Docker using Docker Compose and following the respective tutorial:

https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/docker.html#flink-with-docker-compose

The default job manager port (i.e., 8081) is in use on my host so the cluster fails to launch.

I tried to change the configuration from the respective service definition on docker-compose.yml (e.g., services -> jobmanager -> ports: "5000:5000") with no success; job manager container seems to launch on the default port.

Can anybody help me to proceed with this?

Thanks in advance,

Konstantinos

P.S.: My docker-compose.yml is the following:

version: "2.1"

services:

jobmanager:

    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}

    expose:

      - "6123"

    ports:

      - "5000:5000"

    command: jobmanager

    environment:

      - JOB_MANAGER_RPC_ADDRESS=jobmanager

taskmanager:

    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}

    expose:

      - "6121"

      - "6122"

    depends_on:

      - jobmanager

    command: taskmanager

    links:

      - "jobmanager:jobmanager"

    environment:

      - JOB_MANAGER_RPC_ADDRESS=jobmanager

Pritam Sadhukhan

Re: Unable to change job manager port when launching session cluster on Docker

The problem as I understand is your system port 8081 is already in use, so you want to bind a different port of local system to the container's 8081 port.

Please use <desired-port>:8081 in your docker compose to map your local port to container port.

Else, you may edit your /opt/flink/conf/flink-conf.yaml to change the rest.port and rest.bind-port to the desired port say 9000. Edit your docker-compose 9000:9000 and restart the container again.

Please let me know your observations.

Regards,

Pritam.

On Mon, 21 Oct 2019 at 13:41, Papadopoulos, Konstantinos <[hidden email]> wrote:

Hi Aleksey,

I tried using "8081:5000" as port binding configuration with no success. I also tried different port numbers (i.e,. other than 5000) to bind, but admin seems not to launch.

Is there any easy way to change flink-conf.yaml or pass additional command line argument keeping the docker-compose approach? Or alternatively, should I pull the image, create the container and modify the respective configuration after connecting to it?

Regards,

Konstantinos

From: Aleksey Pak <[hidden email]>
Sent: Friday, October 18, 2019 10:46 PM
To: Papadopoulos, Konstantinos <[hidden email]>
Cc: [hidden email]
Subject: Re: Unable to change job manager port when launching session cluster on Docker

Hi Konstantinos,

Can you try using "8081:5000" as port binding configuration?

This should bind container's 8081 port to 5000 port on host.

If you want to use 5000 port as JobManager's port in the *container*, you would need to change flink-conf.yaml or pass additional command line argument (to override the corresponding config option).

Regards,

Aleksey

On Fri, Oct 18, 2019 at 2:55 PM Papadopoulos, Konstantinos <[hidden email]> wrote:

Hello all,

I am trying to launch an Apache Flink session cluster on Docker using Docker Compose and following the respective tutorial:

https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/docker.html#flink-with-docker-compose

The default job manager port (i.e., 8081) is in use on my host so the cluster fails to launch.

I tried to change the configuration from the respective service definition on docker-compose.yml (e.g., services -> jobmanager -> ports: "5000:5000") with no success; job manager container seems to launch on the default port.

Can anybody help me to proceed with this?

Thanks in advance,

Konstantinos

P.S.: My docker-compose.yml is the following:

version: "2.1"

services:

jobmanager:

    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}

    expose:

      - "6123"

    ports:

      - "5000:5000"

    command: jobmanager

    environment:

      - JOB_MANAGER_RPC_ADDRESS=jobmanager

taskmanager:

    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}

    expose:

      - "6121"

      - "6122"

    depends_on:

      - jobmanager

    command: taskmanager

    links:

      - "jobmanager:jobmanager"

    environment:

      - JOB_MANAGER_RPC_ADDRESS=jobmanager

Papadopoulos, Konstantinos

RE: Unable to change job manager port when launching session cluster on Docker

Hi Pritam,

I tried using <desired-port>:8081 in my docker compose to map your local port to container port.

Session cluster launched successfully. It was my misunderstanding in docker compose port binding sequence since I believed that the first port is the container port while the second one the host.

So, problem solved. Moreover, I tried to mount conf volume and changing the REST port. It also worked, but I will keep the correct port binding approach.

Thanks both for your support.

Regards,

Konstantinos

From: Pritam Sadhukhan <[hidden email]>
Sent: Monday, October 21, 2019 12:10 PM
To: Papadopoulos, Konstantinos <[hidden email]>
Cc: Aleksey Pak <[hidden email]>; [hidden email]
Subject: Re: Unable to change job manager port when launching session cluster on Docker

The problem as I understand is your system port 8081 is already in use, so you want to bind a different port of local system to the container's 8081 port.

Please use <desired-port>:8081 in your docker compose to map your local port to container port.

Else, you may edit your /opt/flink/conf/flink-conf.yaml to change the rest.port and rest.bind-port to the desired port say 9000. Edit your docker-compose 9000:9000 and restart the container again.

Please let me know your observations.

Regards,

Pritam.

On Mon, 21 Oct 2019 at 13:41, Papadopoulos, Konstantinos <[hidden email]> wrote:

Hi Aleksey,

I tried using "8081:5000" as port binding configuration with no success. I also tried different port numbers (i.e,. other than 5000) to bind, but admin seems not to launch.

Is there any easy way to change flink-conf.yaml or pass additional command line argument keeping the docker-compose approach? Or alternatively, should I pull the image, create the container and modify the respective configuration after connecting to it?

Regards,

Konstantinos

From: Aleksey Pak <[hidden email]>
Sent: Friday, October 18, 2019 10:46 PM
To: Papadopoulos, Konstantinos <[hidden email]>
Cc: [hidden email]
Subject: Re: Unable to change job manager port when launching session cluster on Docker

Hi Konstantinos,

Can you try using "8081:5000" as port binding configuration?

This should bind container's 8081 port to 5000 port on host.

If you want to use 5000 port as JobManager's port in the *container*, you would need to change flink-conf.yaml or pass additional command line argument (to override the corresponding config option).

Regards,

Aleksey

On Fri, Oct 18, 2019 at 2:55 PM Papadopoulos, Konstantinos <[hidden email]> wrote:

Hello all,

I am trying to launch an Apache Flink session cluster on Docker using Docker Compose and following the respective tutorial:

https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/docker.html#flink-with-docker-compose

The default job manager port (i.e., 8081) is in use on my host so the cluster fails to launch.

I tried to change the configuration from the respective service definition on docker-compose.yml (e.g., services -> jobmanager -> ports: "5000:5000") with no success; job manager container seems to launch on the default port.

Can anybody help me to proceed with this?

Thanks in advance,

Konstantinos

P.S.: My docker-compose.yml is the following:

version: "2.1"

services:

jobmanager:

    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}

    expose:

      - "6123"

    ports:

      - "5000:5000"

    command: jobmanager

    environment:

      - JOB_MANAGER_RPC_ADDRESS=jobmanager

taskmanager:

    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}

    expose:

      - "6121"

      - "6122"

    depends_on:

      - jobmanager

    command: taskmanager

    links:

      - "jobmanager:jobmanager"

    environment:

      - JOB_MANAGER_RPC_ADDRESS=jobmanager