Container JMX port setting / discovery for Flink on YARN

[Yury Ruchin]

Hi,

I want to enable JMX for my Flink streaming app running in YARN session. How can I specify which ports containers will listen to? If I cannot control it (e. g. they will be chosen randomly) - how can I detect which ports were picked by containers: inspecting logs, looking at the Web UI etc.?

Example: in Apache Storm it is easy to derive JMX port numbers from worker port (slot) numbers and pass it as a JVM argument to the worker start command, so that every slot is always associated with a stable and well-known JMX port.

Any clue is appreciated. Thanks!

[Stefan Richter]

Hi,

can you try adding the following to your flink.yaml?

env.java.opts: -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Best,
Stefan


> Am 24.11.2016 um 16:47 schrieb Yury Ruchin <[hidden email]>:
>
> Hi,
>
> I want to enable JMX for my Flink streaming app running in YARN session. How can I specify which ports containers will listen to? If I cannot control it (e. g. they will be chosen randomly) - how can I detect which ports were picked by containers: inspecting logs, looking at the Web UI etc.?
>
> Example: in Apache Storm it is easy to derive JMX port numbers from worker port (slot) numbers and pass it as a JVM argument to the worker start command, so that every slot is always associated with a stable and well-known JMX port.
>
> Any clue is appreciated. Thanks!

[Yury Ruchin]

Thanks Stefan! I think this would help if I had just one container per node. But that's not my case - there are multiple TaskManagers running on the same node, so setting the same value will likely result in port conflict.

2016-11-25 12:28 GMT+03:00 Stefan Richter <[hidden email]>:

Hi,

can you try adding the following to your flink.yaml?

env.java.opts: -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Best,
Stefan

> Am 24.11.2016 um 16:47 schrieb Yury Ruchin <[hidden email]>:
>
> Hi,
>
> I want to enable JMX for my Flink streaming app running in YARN session. How can I specify which ports containers will listen to? If I cannot control it (e. g. they will be chosen randomly) - how can I detect which ports were picked by containers: inspecting logs, looking at the Web UI etc.?
>
> Example: in Apache Storm it is easy to derive JMX port numbers from worker port (slot) numbers and pass it as a JVM argument to the worker start command, so that every slot is always associated with a stable and well-known JMX port.
>
> Any clue is appreciated. Thanks!

[rmetzger0]

Hi Yury,

Flink is using its own JMX server instance (not the JVM's one). Therefore, you can configure the server yourself.

Check out this documentation page: https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html#reporter

metrics.reporter.my_jmx_reporter.class: org.apache.flink.metrics.jmx.JMXReporter
metrics.reporter.my_jmx_reporter.port: 9020-9040

Flink will print the port it is using in the end into the log of each TaskManager.

On Fri, Nov 25, 2016 at 10:53 AM, Yury Ruchin <[hidden email]> wrote:

Thanks Stefan! I think this would help if I had just one container per node. But that's not my case - there are multiple TaskManagers running on the same node, so setting the same value will likely result in port conflict.

2016-11-25 12:28 GMT+03:00 Stefan Richter <[hidden email]>:

Hi,

can you try adding the following to your flink.yaml?

env.java.opts: -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Best,
Stefan

> Am 24.11.2016 um 16:47 schrieb Yury Ruchin <[hidden email]>:
>
> Hi,
>
> I want to enable JMX for my Flink streaming app running in YARN session. How can I specify which ports containers will listen to? If I cannot control it (e. g. they will be chosen randomly) - how can I detect which ports were picked by containers: inspecting logs, looking at the Web UI etc.?
>
> Example: in Apache Storm it is easy to derive JMX port numbers from worker port (slot) numbers and pass it as a JVM argument to the worker start command, so that every slot is always associated with a stable and well-known JMX port.
>
> Any clue is appreciated. Thanks!

[Yury Ruchin]

Thanks Robert, it works like a charm.

2016-11-25 12:55 GMT+03:00 Robert Metzger <[hidden email]>:

Hi Yury,

Flink is using its own JMX server instance (not the JVM's one). Therefore, you can configure the server yourself.

Check out this documentation page: https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html#reporter

metrics.reporter.my_jmx_reporter.class: org.apache.flink.metrics.jmx.JMXReporter
metrics.reporter.my_jmx_reporter.port: 9020-9040

Flink will print the port it is using in the end into the log of each TaskManager.

On Fri, Nov 25, 2016 at 10:53 AM, Yury Ruchin <[hidden email]> wrote:

Thanks Stefan! I think this would help if I had just one container per node. But that's not my case - there are multiple TaskManagers running on the same node, so setting the same value will likely result in port conflict.

2016-11-25 12:28 GMT+03:00 Stefan Richter <[hidden email]>:

Hi,

can you try adding the following to your flink.yaml?

env.java.opts: -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Best,
Stefan

> Am 24.11.2016 um 16:47 schrieb Yury Ruchin <[hidden email]>:
>
> Hi,
>
> I want to enable JMX for my Flink streaming app running in YARN session. How can I specify which ports containers will listen to? If I cannot control it (e. g. they will be chosen randomly) - how can I detect which ports were picked by containers: inspecting logs, looking at the Web UI etc.?
>
> Example: in Apache Storm it is easy to derive JMX port numbers from worker port (slot) numbers and pass it as a JVM argument to the worker start command, so that every slot is always associated with a stable and well-known JMX port.
>
> Any clue is appreciated. Thanks!