Re: Re: IOException when using Prometheus Monitor

Posted by on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/IOException-when-using-Prometheus-Monitor-tp31094p31096.html

Hi Chesnay,

Although there's Exception, actually the metrics has been put to the pushgateway successfully. 

Promethues version i used:

prometheus-2.8.1.linux-amd64.tar.gz

pushgateway-1.0.0.linux-amd64.tar.gz

flink-metrics-prometheus_2.12-1.8.2.jar

I just download the tar.gz file to CentOS node,  tar -xzvf and then start the process.

Thanks,

Lei

---

[hidden email]

 

From: [hidden email]

Date: 2019-11-20 17:46

To: [hidden email]; [hidden email]

Subject: Re: IOException when using Prometheus Monitor

From what I found so far this appears to be an incompatibility between the pushgateway and client version.

So you can either

a) use an older version of the pushgateway

b) bump the version of the prometheus reporter.

Unfortunately I cannot tell you which version you would need.

On 20/11/2019 10:24, [hidden email] wrote:

Standalone session  flink, 1.8.2 version.

Install prometheus and pushgateway on the same host.

After start-cluster.sh,  every 10 seconds there's following error log:

2019-11-20 17:23:12,849 WARN  org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter  - Failed to push metrics to PushGateway with jobName flink58c234884dcfb60a860d7a040aab6a4a.

java.io.IOException: Response code from http://10.44.51.23:9091/metrics/job/flink58c234884dcfb60a860d7a040aab6a4a was 200

        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)

        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:105)

        at org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:76)

        at org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:430)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Any insight on this?

Thanks,

Lei

---

[hidden email]