Flink and Prometheus setup in K8s

[Wouter Zorgdrager-2]

Hey all,

I'm working on a deployment setup with Flink and Prometheus on Kubernetes. I'm running into the following issues:

1) Is it possible to use the default Flink Docker image [1] and enable the Prometheus reporter? Modifying the flink-config.yaml is easy, but somehow the Prometheus reporter jar needs to be moved within the image. This is easy if use my own Dockerfile (as done here [2]) , but I prefer using the official one.

2) I can define the jobmanager/taskmanager metric endpoints statically, but w.r.t. scaling I prefer to have these resolved/discovered dynamically. Did anyone get a working setup on this? I came across this resource for YARN [3], is there something similar for Kubernetes? Or are there any other ways of configuring Prometheus to pick this up automatically? 

Thanks a lot for your help!

Kind regards,

Wouter

[1]: https://hub.docker.com/_/flink/

[2]: https://github.com/mbode/flink-prometheus-example/blob/master/Dockerfile

[3]: https://github.com/eastcirclek/flink-service-discovery 

[Yun Tang]

Hi Wouter

I have no idea of question-2. But for question-1, you could try to add your steps which already included in your https://github.com/mbode/flink-prometheus-example/blob/master/Dockerfile 's "RUN" phase to your k8s deployment-yaml's "command" phase before launch the cluster in k8s.

Best

Yun Tang

---

From: Wouter Zorgdrager <[hidden email]>
Sent: Monday, May 13, 2019 20:16
To: user
Subject: Flink and Prometheus setup in K8s

 

Hey all,

I'm working on a deployment setup with Flink and Prometheus on Kubernetes. I'm running into the following issues:

1) Is it possible to use the default Flink Docker image [1] and enable the Prometheus reporter? Modifying the flink-config.yaml is easy, but somehow the Prometheus reporter jar needs to be moved within the image. This is easy if use my own Dockerfile (as done here [2]) , but I prefer using the official one.

2) I can define the jobmanager/taskmanager metric endpoints statically, but w.r.t. scaling I prefer to have these resolved/discovered dynamically. Did anyone get a working setup on this? I came across this resource for YARN [3], is there something similar for Kubernetes? Or are there any other ways of configuring Prometheus to pick this up automatically? 

Thanks a lot for your help!

Kind regards,

Wouter

[1]: https://hub.docker.com/_/flink/

[2]: https://github.com/mbode/flink-prometheus-example/blob/master/Dockerfile

[3]: https://github.com/eastcirclek/flink-service-discovery 

[Wouter Zorgdrager-2]

Hi all,

To answer my own questions I worked on the following solution:

1) Custom Docker image which pulls the Flink image and moves Prometheus jar to the correct folder [1, 2]. 

2) I wrote manifests for Kubernetes with service discovery configuration for Kubernetes [3]. Besides the 'official' Flink Kubernetes manifests, I added a TM service which exposes all TM metrics locations so that Prometheus can scrape it. This means that (re)-scaling Flink TM's are automatically picked up by Prometheus. The repository also includes a Grafana setup with a simple dashboard. 

I thought this might be useful for other users!

Cheers,

Wouter

[1]: https://github.com/wzorgdrager/flink-k8s/blob/master/docker/Dockerfile

[2]: https://hub.docker.com/r/wzorgdrager/flink-prometheus

[3]: https://github.com/wzorgdrager/flink-k8s

Op ma 13 mei 2019 om 14:16 schreef Wouter Zorgdrager <[hidden email]>:

Hey all,

I'm working on a deployment setup with Flink and Prometheus on Kubernetes. I'm running into the following issues:

1) Is it possible to use the default Flink Docker image [1] and enable the Prometheus reporter? Modifying the flink-config.yaml is easy, but somehow the Prometheus reporter jar needs to be moved within the image. This is easy if use my own Dockerfile (as done here [2]) , but I prefer using the official one.

2) I can define the jobmanager/taskmanager metric endpoints statically, but w.r.t. scaling I prefer to have these resolved/discovered dynamically. Did anyone get a working setup on this? I came across this resource for YARN [3], is there something similar for Kubernetes? Or are there any other ways of configuring Prometheus to pick this up automatically? 

Thanks a lot for your help!

Kind regards,

Wouter

[1]: https://hub.docker.com/_/flink/

[2]: https://github.com/mbode/flink-prometheus-example/blob/master/Dockerfile

[3]: https://github.com/eastcirclek/flink-service-discovery 

[Fabian Hueske-2]

Thanks for sharing your solution Wouter!

Best, Fabian

Am Mi., 15. Mai 2019 um 15:28 Uhr schrieb Wouter Zorgdrager <[hidden email]>:

Hi all,

To answer my own questions I worked on the following solution:

1) Custom Docker image which pulls the Flink image and moves Prometheus jar to the correct folder [1, 2]. 

2) I wrote manifests for Kubernetes with service discovery configuration for Kubernetes [3]. Besides the 'official' Flink Kubernetes manifests, I added a TM service which exposes all TM metrics locations so that Prometheus can scrape it. This means that (re)-scaling Flink TM's are automatically picked up by Prometheus. The repository also includes a Grafana setup with a simple dashboard. 

I thought this might be useful for other users!

Cheers,

Wouter

[1]: https://github.com/wzorgdrager/flink-k8s/blob/master/docker/Dockerfile

[2]: https://hub.docker.com/r/wzorgdrager/flink-prometheus

[3]: https://github.com/wzorgdrager/flink-k8s

Op ma 13 mei 2019 om 14:16 schreef Wouter Zorgdrager <[hidden email]>:

Hey all,

I'm working on a deployment setup with Flink and Prometheus on Kubernetes. I'm running into the following issues:

1) Is it possible to use the default Flink Docker image [1] and enable the Prometheus reporter? Modifying the flink-config.yaml is easy, but somehow the Prometheus reporter jar needs to be moved within the image. This is easy if use my own Dockerfile (as done here [2]) , but I prefer using the official one.

2) I can define the jobmanager/taskmanager metric endpoints statically, but w.r.t. scaling I prefer to have these resolved/discovered dynamically. Did anyone get a working setup on this? I came across this resource for YARN [3], is there something similar for Kubernetes? Or are there any other ways of configuring Prometheus to pick this up automatically? 

Thanks a lot for your help!

Kind regards,

Wouter

[1]: https://hub.docker.com/_/flink/

[2]: https://github.com/mbode/flink-prometheus-example/blob/master/Dockerfile

[3]: https://github.com/eastcirclek/flink-service-discovery