Apache Flink - Flink Metrics - How to distinguish b/w metrics for two job manager on the same host

[M Singh]

Hi:

I am using AWS EMR with Flink application and two of the job managers are running on the same host.  I am looking at the metrics documentation (Apache Flink 1.9 Documentation: Metrics) and and see the following: 

Apache Flink 1.9 Documentation: Metrics

• metrics.scope.jm
  • Default: <host>.jobmanager
  • Applied to all metrics that were scoped to a job manager.

...

List of all Variables

• JobManager: <host>
• TaskManager: <host>, <tm_id>
• Job: <job_id>, <job_name>
• Task: <task_id>, <task_name>, <task_attempt_id>, <task_attempt_num>, <subtask_index>
• Operator: <operator_id>,<operator_name>, <subtask_index>

My question is there a way to distinguish b/w the two job managers ? I see only the <host> variable for JobManager and since the two are running on the same host, the value is the same.  Is there any other variable that I can use to distinguish the two.

For taskmanager I have taskmanager id but am not sure about the job manager.

Thanks

Mans

[vino yang]

Hi Mans,

IMO, one job manager represents one Flink cluster and one Flink cluster has a suite of Flink configuration e.g. metrics reporter.

Some metrics reporters support tag feature, you can specify it to distinguish different Flink cluster.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#datadog-orgapacheflinkmetricsdatadogdatadoghttpreporter

Best,

Vino

M Singh <[hidden email]> 于2019年12月19日周四 上午2:54写道：

Hi:

I am using AWS EMR with Flink application and two of the job managers are running on the same host.  I am looking at the metrics documentation (Apache Flink 1.9 Documentation: Metrics) and and see the following: 

Apache Flink 1.9 Documentation: Metrics

• metrics.scope.jm
  • Default: <host>.jobmanager
  • Applied to all metrics that were scoped to a job manager.

...

List of all Variables

• JobManager: <host>
• TaskManager: <host>, <tm_id>
• Job: <job_id>, <job_name>
• Task: <task_id>, <task_name>, <task_attempt_id>, <task_attempt_num>, <subtask_index>
• Operator: <operator_id>,<operator_name>, <subtask_index>

My question is there a way to distinguish b/w the two job managers ? I see only the <host> variable for JobManager and since the two are running on the same host, the value is the same.  Is there any other variable that I can use to distinguish the two.

For taskmanager I have taskmanager id but am not sure about the job manager.

Thanks

Mans

[Biao Liu]

Hi Mans,

That's indeed a problem. We have a plan to fix it. I think it could be included in 1.11. You could follow this issue [1] to check the progress.

[1] https://issues.apache.org/jira/browse/FLINK-9543

On Thu, 19 Dec 2019 at 14:51, vino yang <[hidden email]> wrote:

Hi Mans,

IMO, one job manager represents one Flink cluster and one Flink cluster has a suite of Flink configuration e.g. metrics reporter.

Some metrics reporters support tag feature, you can specify it to distinguish different Flink cluster.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#datadog-orgapacheflinkmetricsdatadogdatadoghttpreporter

Best,

Vino

M Singh <[hidden email]> 于2019年12月19日周四 上午2:54写道：

Hi:

I am using AWS EMR with Flink application and two of the job managers are running on the same host.  I am looking at the metrics documentation (Apache Flink 1.9 Documentation: Metrics) and and see the following: 

Apache Flink 1.9 Documentation: Metrics

• metrics.scope.jm
  • Default: <host>.jobmanager
  • Applied to all metrics that were scoped to a job manager.

...

List of all Variables

• JobManager: <host>
• TaskManager: <host>, <tm_id>
• Job: <job_id>, <job_name>
• Task: <task_id>, <task_name>, <task_attempt_id>, <task_attempt_num>, <subtask_index>
• Operator: <operator_id>,<operator_name>, <subtask_index>

My question is there a way to distinguish b/w the two job managers ? I see only the <host> variable for JobManager and since the two are running on the same host, the value is the same.  Is there any other variable that I can use to distinguish the two.

For taskmanager I have taskmanager id but am not sure about the job manager.

Thanks

Mans

[M Singh]

Thanks Vino and Biao for your help.  Mans

On Thursday, December 19, 2019, 02:25:40 AM EST, Biao Liu <[hidden email]> wrote:

Hi Mans,

That's indeed a problem. We have a plan to fix it. I think it could be included in 1.11. You could follow this issue [1] to check the progress.

[1] https://issues.apache.org/jira/browse/FLINK-9543

On Thu, 19 Dec 2019 at 14:51, vino yang <[hidden email]> wrote:

Hi Mans,

IMO, one job manager represents one Flink cluster and one Flink cluster has a suite of Flink configuration e.g. metrics reporter.

Some metrics reporters support tag feature, you can specify it to distinguish different Flink cluster.[1]

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#datadog-orgapacheflinkmetricsdatadogdatadoghttpreporter

Best,

Vino

M Singh <[hidden email]> 于2019年12月19日周四 上午2:54写道：

Hi:

I am using AWS EMR with Flink application and two of the job managers are running on the same host.  I am looking at the metrics documentation (Apache Flink 1.9 Documentation: Metrics) and and see the following: 

Apache Flink 1.9 Documentation: Metrics

• metrics.scope.jm
  • Default: <host>.jobmanager
  • Applied to all metrics that were scoped to a job manager.

...

List of all Variables

• JobManager: <host>
• TaskManager: <host>, <tm_id>
• Job: <job_id>, <job_name>
• Task: <task_id>, <task_name>, <task_attempt_id>, <task_attempt_num>, <subtask_index>
• Operator: <operator_id>,<operator_name>, <subtask_index>

My question is there a way to distinguish b/w the two job managers ? I see only the <host> variable for JobManager and since the two are running on the same host, the value is the same.  Is there any other variable that I can use to distinguish the two.

For taskmanager I have taskmanager id but am not sure about the job manager.

Thanks

Mans