Kafka Consumer Metrics For Help

[Yunhui Han]

Hi, all 

I use influxdb to monitor flink metrics. However, I am confused with the metric taskmanager job_task_operator_KafkaConsumer_count. What does it mean? Is it OK for that the value is around 150 ?

Your Sincerely

[Dawid Wysakowicz-2]

Hi,

Which version of Flink, Kafka and Kafka connector are you using? The KafkaConsumer infix means it is most probably a metric forwarded from the Kafka client. (I could not find one, named like that, so a concrete versions of different components could help).

Do you register any custom metrics in a TimestampAssigner or WatermarkGenerator?

Best,

Dawid

On 15/09/2020 09:44, Yunhui Han wrote:

Hi, all 

I use influxdb to monitor flink metrics. However, I am confused with the metric taskmanager job_task_operator_KafkaConsumer_count. What does it mean? Is it OK for that the value is around 150 ?

Your Sincerely

[Yunhui Han]

Hi，Dawid

Thank you for your response. 

The version of Flink is flink-1.10.1, and the Kafka version is 0.10. The connector version is flink-connector-kafka-0.10_2.11.

This metric is generated by flink itself. I did not register any custom metrics.

Best

On Tue, Sep 15, 2020 at 8:16 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Which version of Flink, Kafka and Kafka connector are you using? The KafkaConsumer infix means it is most probably a metric forwarded from the Kafka client. (I could not find one, named like that, so a concrete versions of different components could help).

Do you register any custom metrics in a TimestampAssigner or WatermarkGenerator?

Best,

Dawid

On 15/09/2020 09:44, Yunhui Han wrote:

Hi, all 

I use influxdb to monitor flink metrics. However, I am confused with the metric taskmanager job_task_operator_KafkaConsumer_count. What does it mean? Is it OK for that the value is around 150 ?

Your Sincerely

[Dawid Wysakowicz-2]

As I said the infix "KafkaConsumer" means it is a metric forwarded from Kafka client itself. (most probably there are just a few exceptions). This one in particular is the count of registered metrics[1].

metricName("count", "kafka-metrics-count", "total number of registered metrics")

Best,

Dawid

[1] https://issues.apache.org/jira/browse/KAFKA-2668

On 16/09/2020 03:33, Yunhui Han wrote:

Hi，Dawid

Thank you for your response. 

The version of Flink is flink-1.10.1, and the Kafka version is 0.10. The connector version is flink-connector-kafka-0.10_2.11.

This metric is generated by flink itself. I did not register any custom metrics.

Best

On Tue, Sep 15, 2020 at 8:16 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Which version of Flink, Kafka and Kafka connector are you using? The KafkaConsumer infix means it is most probably a metric forwarded from the Kafka client. (I could not find one, named like that, so a concrete versions of different components could help).

Do you register any custom metrics in a TimestampAssigner or WatermarkGenerator?

Best,

Dawid

On 15/09/2020 09:44, Yunhui Han wrote:

Hi, all 

I use influxdb to monitor flink metrics. However, I am confused with the metric taskmanager job_task_operator_KafkaConsumer_count. What does it mean? Is it OK for that the value is around 150 ?

Your Sincerely