(DEPRECATED) Apache Flink User Mailing List archive.
Login  Register

IOException when using Prometheus Monitor

classic Classic list List threaded Threaded
♦
♦
 4 messages Options Options
 
Embed post
Permalink
wanglei2@geekplus.com.cn
Reply | Threaded
Open this post in threaded view
♦
♦
| More
Print post
Permalink
Nov 20, 2019; 9:24am

IOException when using Prometheus Monitor

wanglei2@geekplus.com.cn
Standalone session  flink, 1.8.2 version.
Install prometheus and pushgateway on the same host.

After start-cluster.sh,  every 10 seconds there's following error log:
2019-11-20 17:23:12,849 WARN  org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter  - Failed to push metrics to PushGateway with jobName flink58c234884dcfb60a860d7a040aab6a4a.
java.io.IOException: Response code from http://10.44.51.23:9091/metrics/job/flink58c234884dcfb60a860d7a040aab6a4a was 200
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:105)
        at org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:76)
        at org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


Any insight on this?

Thanks,
Lei
Chesnay Schepler
Reply | Threaded
Open this post in threaded view
♦
♦
| More
Print post
Permalink
Nov 20, 2019; 9:46am

Re: IOException when using Prometheus Monitor

Chesnay Schepler
From what I found so far this appears to be an incompatibility between the pushgateway and client version.

So you can either
a) use an older version of the pushgateway
b) bump the version of the prometheus reporter.

Unfortunately I cannot tell you which version you would need.

On 20/11/2019 10:24, [hidden email] wrote:
Standalone session  flink, 1.8.2 version.
Install prometheus and pushgateway on the same host.

After start-cluster.sh,  every 10 seconds there's following error log:
2019-11-20 17:23:12,849 WARN  org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter  - Failed to push metrics to PushGateway with jobName flink58c234884dcfb60a860d7a040aab6a4a.
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:105)
        at org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:76)
        at org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


Any insight on this?

Thanks,
Lei


... [show rest of quote]
wanglei2@geekplus.com.cn
Reply | Threaded
Open this post in threaded view
♦
♦
| More
Print post
Permalink
Nov 20, 2019; 10:19am

Re: Re: IOException when using Prometheus Monitor

wanglei2@geekplus.com.cn
Hi Chesnay,

Although there's Exception, actually the metrics has been put to the pushgateway successfully. 

Promethues version i used:

prometheus-2.8.1.linux-amd64.tar.gz
pushgateway-1.0.0.linux-amd64.tar.gz
flink-metrics-prometheus_2.12-1.8.2.jar

I just download the tar.gz file to CentOS node,  tar -xzvf and then start the process.

Thanks,
Lei



 
Date: 2019-11-20 17:46
Subject: Re: IOException when using Prometheus Monitor
From what I found so far this appears to be an incompatibility between the pushgateway and client version.

So you can either
a) use an older version of the pushgateway
b) bump the version of the prometheus reporter.

Unfortunately I cannot tell you which version you would need.

On 20/11/2019 10:24, [hidden email] wrote:
Standalone session  flink, 1.8.2 version.
Install prometheus and pushgateway on the same host.

After start-cluster.sh,  every 10 seconds there's following error log:
2019-11-20 17:23:12,849 WARN  org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter  - Failed to push metrics to PushGateway with jobName flink58c234884dcfb60a860d7a040aab6a4a.
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:105)
        at org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:76)
        at org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


Any insight on this?

Thanks,
Lei


... [show rest of quote]


... [show rest of quote]
wanglei2@geekplus.com.cn
Reply | Threaded
Open this post in threaded view
♦
♦
| More
Print post
Permalink
Nov 20, 2019; 10:38am

Re: Re: IOException when using Prometheus Monitor

wanglei2@geekplus.com.cn
After downgrade the  pushgateway to pushgateway-0.8.0.linux-amd64.tar.gz, no this Exception again.

Thanks very much.

 
Date: 2019-11-20 18:19
Subject: Re: Re: IOException when using Prometheus Monitor
Hi Chesnay,

Although there's Exception, actually the metrics has been put to the pushgateway successfully. 

Promethues version i used:

prometheus-2.8.1.linux-amd64.tar.gz
pushgateway-1.0.0.linux-amd64.tar.gz
flink-metrics-prometheus_2.12-1.8.2.jar

I just download the tar.gz file to CentOS node,  tar -xzvf and then start the process.

Thanks,
Lei



 
Date: 2019-11-20 17:46
Subject: Re: IOException when using Prometheus Monitor
From what I found so far this appears to be an incompatibility between the pushgateway and client version.

So you can either
a) use an older version of the pushgateway
b) bump the version of the prometheus reporter.

Unfortunately I cannot tell you which version you would need.

On 20/11/2019 10:24, [hidden email] wrote:
Standalone session  flink, 1.8.2 version.
Install prometheus and pushgateway on the same host.

After start-cluster.sh,  every 10 seconds there's following error log:
2019-11-20 17:23:12,849 WARN  org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter  - Failed to push metrics to PushGateway with jobName flink58c234884dcfb60a860d7a040aab6a4a.
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)
        at org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:105)
        at org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:76)
        at org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


Any insight on this?

Thanks,
Lei


... [show rest of quote]


... [show rest of quote]
... [show rest of quote]
Free forum by Nabble Edit this page