(DEPRECATED) Apache Flink User Mailing List archive.

IOException when using Prometheus Monitor

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
wanglei2@geekplus.com.cn
Reply | Threaded
Open this post in threaded view
♦
♦
|

IOException when using Prometheus Monitor

wanglei2@geekplus.com.cn
Standalone session  flink, 1.8.2 version.
Install prometheus and pushgateway on the same host.

After start-cluster.sh,  every 10 seconds there's following error log:
2019-11-20 17:23:12,849 WARN  org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter  - Failed to push metrics to PushGateway with jobName flink58c234884dcfb60a860d7a040aab6a4a.
java.io.IOException: Response code from http://10.44.51.23:9091/metrics/job/flink58c234884dcfb60a860d7a040aab6a4a was 200
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:105)
        at org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:76)
        at org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


Any insight on this?

Thanks,
Lei
Chesnay Schepler
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: IOException when using Prometheus Monitor

Chesnay Schepler
From what I found so far this appears to be an incompatibility between the pushgateway and client version.

So you can either
a) use an older version of the pushgateway
b) bump the version of the prometheus reporter.

Unfortunately I cannot tell you which version you would need.

On 20/11/2019 10:24, [hidden email] wrote:
Standalone session  flink, 1.8.2 version.
Install prometheus and pushgateway on the same host.

After start-cluster.sh,  every 10 seconds there's following error log:
2019-11-20 17:23:12,849 WARN  org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter  - Failed to push metrics to PushGateway with jobName flink58c234884dcfb60a860d7a040aab6a4a.
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:105)
        at org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:76)
        at org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


Any insight on this?

Thanks,
Lei
wanglei2@geekplus.com.cn
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Re: IOException when using Prometheus Monitor

wanglei2@geekplus.com.cn
Hi Chesnay,

Although there's Exception, actually the metrics has been put to the pushgateway successfully. 

Promethues version i used:

prometheus-2.8.1.linux-amd64.tar.gz
pushgateway-1.0.0.linux-amd64.tar.gz
flink-metrics-prometheus_2.12-1.8.2.jar

I just download the tar.gz file to CentOS node,  tar -xzvf and then start the process.

Thanks,
Lei



 
Date: 2019-11-20 17:46
Subject: Re: IOException when using Prometheus Monitor
From what I found so far this appears to be an incompatibility between the pushgateway and client version.

So you can either
a) use an older version of the pushgateway
b) bump the version of the prometheus reporter.

Unfortunately I cannot tell you which version you would need.

On 20/11/2019 10:24, [hidden email] wrote:
Standalone session  flink, 1.8.2 version.
Install prometheus and pushgateway on the same host.

After start-cluster.sh,  every 10 seconds there's following error log:
2019-11-20 17:23:12,849 WARN  org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter  - Failed to push metrics to PushGateway with jobName flink58c234884dcfb60a860d7a040aab6a4a.
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:105)
        at org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:76)
        at org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


Any insight on this?

Thanks,
Lei
wanglei2@geekplus.com.cn
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Re: IOException when using Prometheus Monitor

wanglei2@geekplus.com.cn
After downgrade the  pushgateway to pushgateway-0.8.0.linux-amd64.tar.gz, no this Exception again.

Thanks very much.

 
Date: 2019-11-20 18:19
Subject: Re: Re: IOException when using Prometheus Monitor
Hi Chesnay,

Although there's Exception, actually the metrics has been put to the pushgateway successfully. 

Promethues version i used:

prometheus-2.8.1.linux-amd64.tar.gz
pushgateway-1.0.0.linux-amd64.tar.gz
flink-metrics-prometheus_2.12-1.8.2.jar

I just download the tar.gz file to CentOS node,  tar -xzvf and then start the process.

Thanks,
Lei



 
Date: 2019-11-20 17:46
Subject: Re: IOException when using Prometheus Monitor
From what I found so far this appears to be an incompatibility between the pushgateway and client version.

So you can either
a) use an older version of the pushgateway
b) bump the version of the prometheus reporter.

Unfortunately I cannot tell you which version you would need.

On 20/11/2019 10:24, [hidden email] wrote:
Standalone session  flink, 1.8.2 version.
Install prometheus and pushgateway on the same host.

After start-cluster.sh,  every 10 seconds there's following error log:
2019-11-20 17:23:12,849 WARN  org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter  - Failed to push metrics to PushGateway with jobName flink58c234884dcfb60a860d7a040aab6a4a.
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:105)
        at org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:76)
        at org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


Any insight on this?

Thanks,
Lei
Free forum by Nabble Edit this page