Flink PrometheusReporter support for HTTPS

[Ashutosh Uttam]

Hi All,

Does PrometheusReporter provide support for HTTPS?. I couldn't find any information in flink documentation.

Is there any way we can achieve the same?

Thanks & Regards,
Ashutosh

[austin.ce]

Hi Ashutosh,

How are you deploying your Flink apps? Would running a reverse proxy like Nginx or Envoy that handles the HTTPS connection work for you?

Best,

Austin

On Sat, Jun 12, 2021 at 1:11 PM Ashutosh Uttam <[hidden email]> wrote:

Hi All,

Does PrometheusReporter provide support for HTTPS?. I couldn't find any information in flink documentation.

Is there any way we can achieve the same?

Thanks & Regards,
Ashutosh

[Ashutosh Uttam]

Hi Austin,

I am deploying Flink on K8s with multiple Job Manager pods (For HA)  & Task Manager pods.

Each JobManager & Task Manager are running an PrometheusReporter instance and using Prometheus’ service discovery support for Kubernetes to discover all pods (Job Manager & Task Manager) and expose the container as targets

Please let me know if a reverse proxy can work on this deployment as we have multiple JMs & TMs and cannot use static scrape targets

Regards,

Ashutosh

On Sun, Jun 13, 2021 at 2:25 AM Austin Cawley-Edwards <[hidden email]> wrote:

Hi Ashutosh,

How are you deploying your Flink apps? Would running a reverse proxy like Nginx or Envoy that handles the HTTPS connection work for you?

Best,

Austin

On Sat, Jun 12, 2021 at 1:11 PM Ashutosh Uttam <[hidden email]> wrote:

Hi All,

Does PrometheusReporter provide support for HTTPS?. I couldn't find any information in flink documentation.

Is there any way we can achieve the same?

Thanks & Regards,
Ashutosh

[rmetzger0]

It seems like the PrometheusReporter doesn't support HTTPS.

The Flink reporter seems to be based on the HttpServer prometheus client. I wonder if using the servlet client would allow us to add HTTPS support: https://github.com/prometheus/client_java/blob/master/simpleclient_servlet/src/main/java/io/prometheus/client/exporter/MetricsServlet.java / https://github.com/prometheus/client_java#http

Running the Servlet inside an SSL enabled Jetty should do the trick.

If this is reliable, you could consider contributing this back to Flink.

On Sun, Jun 13, 2021 at 4:27 PM Ashutosh Uttam <[hidden email]> wrote:

Hi Austin,

I am deploying Flink on K8s with multiple Job Manager pods (For HA)  & Task Manager pods.

Each JobManager & Task Manager are running an PrometheusReporter instance and using Prometheus’ service discovery support for Kubernetes to discover all pods (Job Manager & Task Manager) and expose the container as targets

Please let me know if a reverse proxy can work on this deployment as we have multiple JMs & TMs and cannot use static scrape targets

Regards,

Ashutosh

On Sun, Jun 13, 2021 at 2:25 AM Austin Cawley-Edwards <[hidden email]> wrote:

Hi Ashutosh,

How are you deploying your Flink apps? Would running a reverse proxy like Nginx or Envoy that handles the HTTPS connection work for you?

Best,

Austin

On Sat, Jun 12, 2021 at 1:11 PM Ashutosh Uttam <[hidden email]> wrote:

Hi All,

Does PrometheusReporter provide support for HTTPS?. I couldn't find any information in flink documentation.

Is there any way we can achieve the same?

Thanks & Regards,
Ashutosh

[austin.ce]

Hi Ashutosh,

Sorry for the delayed response + thanks Robert for the good links and idea.

Alternatively, Flink on K8s is a perfect scenario for running a sidecar proxy or gateway that handles HTTPS connections. The advantage here is that you decouple managing SSL certifications + rotation from your actual Flink applications. If you were to run an HTTPS server directly within Flink, and changes to that server would inherently cause you to modify your running Flink application (downtime). In either of the other solutions, Flink applications stay running independently of "SSL management".

Running a gateway that exposes an HTTPS server, terminates TLS, and forwards traffic to the Flink pods over HTTP is the simpler approach. You will still have HTTP traffic to/from your Flink pods, though this may be acceptable in some situations. For this approach, I would consider either a Kong[1] or Nginx[2] gateway, which both have good K8s support.

The second option would be to introduce a service mesh like Kuma[3] or Istio[4] (disclaimer: I'm a Kuma maintainer), which runs a proxy container next to each of your Flink containers (i.e., on each JobManager and TaskManager). This ensures that HTTPS traffic goes directly to each Pod, but can be a bit more cumbersome to set up though they automate much more of the SSL lifecycle than gateways.

Do those ideas make sense? There is a nice article on the CNCF blog about API Gateways vs. Service Meshes[5], which I've found helpful in the past.

Best,

Austin

[1]: https://github.com/kong/kong

[2]: https://nginx.org/en/docs/http/configuring_https_servers.html

[3]: https://kuma.io/

[4]: https://istio.io/latest/

[5]: https://www.cncf.io/blog/2020/03/06/the-difference-between-api-gateways-and-service-mesh/

On Wed, Jun 16, 2021 at 8:50 AM Robert Metzger <[hidden email]> wrote:

It seems like the PrometheusReporter doesn't support HTTPS.

The Flink reporter seems to be based on the HttpServer prometheus client. I wonder if using the servlet client would allow us to add HTTPS support: https://github.com/prometheus/client_java/blob/master/simpleclient_servlet/src/main/java/io/prometheus/client/exporter/MetricsServlet.java / https://github.com/prometheus/client_java#http

Running the Servlet inside an SSL enabled Jetty should do the trick.

If this is reliable, you could consider contributing this back to Flink.

On Sun, Jun 13, 2021 at 4:27 PM Ashutosh Uttam <[hidden email]> wrote:

Hi Austin,

I am deploying Flink on K8s with multiple Job Manager pods (For HA)  & Task Manager pods.

Each JobManager & Task Manager are running an PrometheusReporter instance and using Prometheus’ service discovery support for Kubernetes to discover all pods (Job Manager & Task Manager) and expose the container as targets

Please let me know if a reverse proxy can work on this deployment as we have multiple JMs & TMs and cannot use static scrape targets

Regards,

Ashutosh

On Sun, Jun 13, 2021 at 2:25 AM Austin Cawley-Edwards <[hidden email]> wrote:

Hi Ashutosh,

How are you deploying your Flink apps? Would running a reverse proxy like Nginx or Envoy that handles the HTTPS connection work for you?

Best,

Austin

On Sat, Jun 12, 2021 at 1:11 PM Ashutosh Uttam <[hidden email]> wrote:

Hi All,

Does PrometheusReporter provide support for HTTPS?. I couldn't find any information in flink documentation.

Is there any way we can achieve the same?

Thanks & Regards,
Ashutosh