[External] Flink 1.7.1 on EMR metrics

[Padarn Wilson]

Hello all,

I am trying to run Flink 1.7.1 on EMR and having some trouble with metric reporting.

I was using the DataDogHttpReporter, but have also tried the StatsDReporter, but with both was seeing no metrics being collected.

To debug this I implemented my own reporter (based on StatsDReporter) and logged the name of the metric being sent:

private void send(final String name, final String value) {
   log.info("STATSD SENDING: ", name, value);
   try {
      String formatted = String.format("%s:%s|g", name, value);
      byte[] data = formatted.getBytes(StandardCharsets.UTF_8);
      socket.send(new DatagramPacket(data, data.length, this.address));
   }
   catch (IOException e) {
      LOG.error("unable to send packet to statsd at '{}:{}'", address.getHostName(), address.getPort());
   }
}

This code is certainly reached, because in my log I see a lot of this:

2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,352 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 

As you can see, the name and value for the metric being reported is empty.

And the logs show everything initialized fine with no error:

2019-05-30 10:18:30,342 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Configuring stsd with {port=8125, host=127.0.0.1, class=com.grab.statsd.StatsDReporter}.
2019-05-30 10:18:30,344 INFO  com.grab.statsd.StatsDReporter                                - Configured StatsDReporter with {host:127.0.0.1, port:8125}
2019-05-30 10:18:30,344 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Periodically reporting metrics in intervals of 10 SECONDS for reporter stsd of type com.grab.statsd.StatsDReporter.

Has anyone else  tried to work with Flink and metrics on EMR 1.7.1 (latest version on EMR). If so, any pointers as to what could be set up incorrectly?

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[Yun Tang]

Hi Padarn

If you want to verify why no metrics sending out, how about using the built-in Slf4j reporter [1] which would record metrics in logs.

If you could view the metrics after enabled slf4j-reporter, you could then compare the configurations.

Best

Yun Tang

[1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter

---

From: Padarn Wilson <[hidden email]>
Sent: Thursday, May 30, 2019 18:20
To: user
Subject: [External] Flink 1.7.1 on EMR metrics

 

Hello all,

I am trying to run Flink 1.7.1 on EMR and having some trouble with metric reporting.

I was using the DataDogHttpReporter, but have also tried the StatsDReporter, but with both was seeing no metrics being collected.

To debug this I implemented my own reporter (based on StatsDReporter) and logged the name of the metric being sent:

private void send(final String name, final String value) {
   log.info("STATSD SENDING: ", name, value);
   try {
      String formatted = String.format("%s:%s|g", name, value);
      byte[] data = formatted.getBytes(StandardCharsets.UTF_8);
      socket.send(new DatagramPacket(data, data.length, this.address));
   }
   catch (IOException e) {
      LOG.error("unable to send packet to statsd at '{}:{}'", address.getHostName(), address.getPort());
   }
}

This code is certainly reached, because in my log I see a lot of this:

2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,352 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 

As you can see, the name and value for the metric being reported is empty.

And the logs show everything initialized fine with no error:

2019-05-30 10:18:30,342 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Configuring stsd with {port=8125, host=127.0.0.1, class=com.grab.statsd.StatsDReporter}.
2019-05-30 10:18:30,344 INFO  com.grab.statsd.StatsDReporter                                - Configured StatsDReporter with {host:127.0.0.1, port:8125}
2019-05-30 10:18:30,344 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Periodically reporting metrics in intervals of 10 SECONDS for reporter stsd of type com.grab.statsd.StatsDReporter.

Has anyone else  tried to work with Flink and metrics on EMR 1.7.1 (latest version on EMR). If so, any pointers as to what could be set up incorrectly?

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[Peter Groesbeck]

Hi Padarn for what it's worth I am using DataDog metrics on EMR with Flink 1.7.1 and this here my flink-conf configuration:

- Classification: flink-conf
  ConfigurationProperties:
    metrics.reporter.dghttp.class: org.apache.flink.metrics.datadog.DatadogHttpReporter
    metrics.reporter.dghttp.apikey: <your-datadog-key>
    metrics.reporter.dghttp.tags: 'service:myservice,env:prod,region:us-east-1'
    metrics.scope.jm: 'jobmanager'
    metrics.scope.jm.job: 'jobmanager'
    metrics.scope.operator: 'taskmanager.<job_name>.<operator_name>'
    metrics.scope.task: 'taskmanager.<job_name>.<task_name>'
    metrics.scope.tm: 'taskmanager'
    metrics.scope.tm.job: 'taskmanager'
  Configurations: []

On Thu, May 30, 2019 at 6:46 AM Yun Tang <[hidden email]> wrote:

Hi Padarn

If you want to verify why no metrics sending out, how about using the built-in Slf4j reporter [1] which would record metrics in logs.

If you could view the metrics after enabled slf4j-reporter, you could then compare the configurations.

Best

Yun Tang

[1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter

---

From: Padarn Wilson <[hidden email]>
Sent: Thursday, May 30, 2019 18:20
To: user
Subject: [External] Flink 1.7.1 on EMR metrics

 

Hello all,

I am trying to run Flink 1.7.1 on EMR and having some trouble with metric reporting.

I was using the DataDogHttpReporter, but have also tried the StatsDReporter, but with both was seeing no metrics being collected.

To debug this I implemented my own reporter (based on StatsDReporter) and logged the name of the metric being sent:

private void send(final String name, final String value) {
   log.info("STATSD SENDING: ", name, value);
   try {
      String formatted = String.format("%s:%s|g", name, value);
      byte[] data = formatted.getBytes(StandardCharsets.UTF_8);
      socket.send(new DatagramPacket(data, data.length, this.address));
   }
   catch (IOException e) {
      LOG.error("unable to send packet to statsd at '{}:{}'", address.getHostName(), address.getPort());
   }
}

This code is certainly reached, because in my log I see a lot of this:

2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,352 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 

As you can see, the name and value for the metric being reported is empty.

And the logs show everything initialized fine with no error:

2019-05-30 10:18:30,342 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Configuring stsd with {port=8125, host=127.0.0.1, class=com.grab.statsd.StatsDReporter}.
2019-05-30 10:18:30,344 INFO  com.grab.statsd.StatsDReporter                                - Configured StatsDReporter with {host:127.0.0.1, port:8125}
2019-05-30 10:18:30,344 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Periodically reporting metrics in intervals of 10 SECONDS for reporter stsd of type com.grab.statsd.StatsDReporter.

Has anyone else  tried to work with Flink and metrics on EMR 1.7.1 (latest version on EMR). If so, any pointers as to what could be set up incorrectly?

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[Padarn Wilson-2]

Thanks both: Using the the inbuilt Slf4j reporter is a great idea, I will do this.

@Peter.Groesbeck - appreciate  the config. This looks very similar to what I had, but if it is working for you perhaps there is something else missing from our EMR setup. Will go back and double check the connectivity from all the instances.

On Thu, May 30, 2019 at 9:42 PM Peter Groesbeck <[hidden email]> wrote:

Hi Padarn for what it's worth I am using DataDog metrics on EMR with Flink 1.7.1 and this here my flink-conf configuration:

- Classification: flink-conf
  ConfigurationProperties:
    metrics.reporter.dghttp.class: org.apache.flink.metrics.datadog.DatadogHttpReporter
    metrics.reporter.dghttp.apikey: <your-datadog-key>
    metrics.reporter.dghttp.tags: 'service:myservice,env:prod,region:us-east-1'
    metrics.scope.jm: 'jobmanager'
    metrics.scope.jm.job: 'jobmanager'
    metrics.scope.operator: 'taskmanager.<job_name>.<operator_name>'
    metrics.scope.task: 'taskmanager.<job_name>.<task_name>'
    metrics.scope.tm: 'taskmanager'
    metrics.scope.tm.job: 'taskmanager'
  Configurations: []

On Thu, May 30, 2019 at 6:46 AM Yun Tang <[hidden email]> wrote:

Hi Padarn

If you want to verify why no metrics sending out, how about using the built-in Slf4j reporter [1] which would record metrics in logs.

If you could view the metrics after enabled slf4j-reporter, you could then compare the configurations.

Best

Yun Tang

[1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter

---

From: Padarn Wilson <[hidden email]>
Sent: Thursday, May 30, 2019 18:20
To: user
Subject: [External] Flink 1.7.1 on EMR metrics

 

Hello all,

I am trying to run Flink 1.7.1 on EMR and having some trouble with metric reporting.

I was using the DataDogHttpReporter, but have also tried the StatsDReporter, but with both was seeing no metrics being collected.

To debug this I implemented my own reporter (based on StatsDReporter) and logged the name of the metric being sent:

private void send(final String name, final String value) {
   log.info("STATSD SENDING: ", name, value);
   try {
      String formatted = String.format("%s:%s|g", name, value);
      byte[] data = formatted.getBytes(StandardCharsets.UTF_8);
      socket.send(new DatagramPacket(data, data.length, this.address));
   }
   catch (IOException e) {
      LOG.error("unable to send packet to statsd at '{}:{}'", address.getHostName(), address.getPort());
   }
}

This code is certainly reached, because in my log I see a lot of this:

2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,352 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 

As you can see, the name and value for the metric being reported is empty.

And the logs show everything initialized fine with no error:

2019-05-30 10:18:30,342 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Configuring stsd with {port=8125, host=127.0.0.1, class=com.grab.statsd.StatsDReporter}.
2019-05-30 10:18:30,344 INFO  com.grab.statsd.StatsDReporter                                - Configured StatsDReporter with {host:127.0.0.1, port:8125}
2019-05-30 10:18:30,344 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Periodically reporting metrics in intervals of 10 SECONDS for reporter stsd of type com.grab.statsd.StatsDReporter.

Has anyone else  tried to work with Flink and metrics on EMR 1.7.1 (latest version on EMR). If so, any pointers as to what could be set up incorrectly?

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[phoenixjiangnan]

To answer your question on your debugging code, your reporter has a bug:

log.info("STATSD SENDING: ", name, value);

should be ->

log.info("STATSD SENDING: {} {}", name, value);

-

On Sat, Jun 1, 2019 at 7:30 PM Padarn Wilson <[hidden email]> wrote:

Thanks both: Using the the inbuilt Slf4j reporter is a great idea, I will do this.

@Peter.Groesbeck - appreciate  the config. This looks very similar to what I had, but if it is working for you perhaps there is something else missing from our EMR setup. Will go back and double check the connectivity from all the instances.

On Thu, May 30, 2019 at 9:42 PM Peter Groesbeck <[hidden email]> wrote:

Hi Padarn for what it's worth I am using DataDog metrics on EMR with Flink 1.7.1 and this here my flink-conf configuration:

- Classification: flink-conf
  ConfigurationProperties:
    metrics.reporter.dghttp.class: org.apache.flink.metrics.datadog.DatadogHttpReporter
    metrics.reporter.dghttp.apikey: <your-datadog-key>
    metrics.reporter.dghttp.tags: 'service:myservice,env:prod,region:us-east-1'
    metrics.scope.jm: 'jobmanager'
    metrics.scope.jm.job: 'jobmanager'
    metrics.scope.operator: 'taskmanager.<job_name>.<operator_name>'
    metrics.scope.task: 'taskmanager.<job_name>.<task_name>'
    metrics.scope.tm: 'taskmanager'
    metrics.scope.tm.job: 'taskmanager'
  Configurations: []

On Thu, May 30, 2019 at 6:46 AM Yun Tang <[hidden email]> wrote:

Hi Padarn

If you want to verify why no metrics sending out, how about using the built-in Slf4j reporter [1] which would record metrics in logs.

If you could view the metrics after enabled slf4j-reporter, you could then compare the configurations.

Best

Yun Tang

[1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter

---

From: Padarn Wilson <[hidden email]>
Sent: Thursday, May 30, 2019 18:20
To: user
Subject: [External] Flink 1.7.1 on EMR metrics

 

Hello all,

I am trying to run Flink 1.7.1 on EMR and having some trouble with metric reporting.

I was using the DataDogHttpReporter, but have also tried the StatsDReporter, but with both was seeing no metrics being collected.

To debug this I implemented my own reporter (based on StatsDReporter) and logged the name of the metric being sent:

private void send(final String name, final String value) {
   log.info("STATSD SENDING: ", name, value);
   try {
      String formatted = String.format("%s:%s|g", name, value);
      byte[] data = formatted.getBytes(StandardCharsets.UTF_8);
      socket.send(new DatagramPacket(data, data.length, this.address));
   }
   catch (IOException e) {
      LOG.error("unable to send packet to statsd at '{}:{}'", address.getHostName(), address.getPort());
   }
}

This code is certainly reached, because in my log I see a lot of this:

2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,352 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 

As you can see, the name and value for the metric being reported is empty.

And the logs show everything initialized fine with no error:

2019-05-30 10:18:30,342 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Configuring stsd with {port=8125, host=127.0.0.1, class=com.grab.statsd.StatsDReporter}.
2019-05-30 10:18:30,344 INFO  com.grab.statsd.StatsDReporter                                - Configured StatsDReporter with {host:127.0.0.1, port:8125}
2019-05-30 10:18:30,344 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Periodically reporting metrics in intervals of 10 SECONDS for reporter stsd of type com.grab.statsd.StatsDReporter.

Has anyone else  tried to work with Flink and metrics on EMR 1.7.1 (latest version on EMR). If so, any pointers as to what could be set up incorrectly?

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[Padarn Wilson-2]

Doh - stupid mistake. Thanks for pointing that out.

On Sun, Jun 2, 2019 at 12:52 PM Bowen Li <[hidden email]> wrote:

To answer your question on your debugging code, your reporter has a bug:

log.info("STATSD SENDING: ", name, value);

should be ->

log.info("STATSD SENDING: {} {}", name, value);

-

On Sat, Jun 1, 2019 at 7:30 PM Padarn Wilson <[hidden email]> wrote:

Thanks both: Using the the inbuilt Slf4j reporter is a great idea, I will do this.

@Peter.Groesbeck - appreciate  the config. This looks very similar to what I had, but if it is working for you perhaps there is something else missing from our EMR setup. Will go back and double check the connectivity from all the instances.

On Thu, May 30, 2019 at 9:42 PM Peter Groesbeck <[hidden email]> wrote:

Hi Padarn for what it's worth I am using DataDog metrics on EMR with Flink 1.7.1 and this here my flink-conf configuration:

- Classification: flink-conf
  ConfigurationProperties:
    metrics.reporter.dghttp.class: org.apache.flink.metrics.datadog.DatadogHttpReporter
    metrics.reporter.dghttp.apikey: <your-datadog-key>
    metrics.reporter.dghttp.tags: 'service:myservice,env:prod,region:us-east-1'
    metrics.scope.jm: 'jobmanager'
    metrics.scope.jm.job: 'jobmanager'
    metrics.scope.operator: 'taskmanager.<job_name>.<operator_name>'
    metrics.scope.task: 'taskmanager.<job_name>.<task_name>'
    metrics.scope.tm: 'taskmanager'
    metrics.scope.tm.job: 'taskmanager'
  Configurations: []

On Thu, May 30, 2019 at 6:46 AM Yun Tang <[hidden email]> wrote:

Hi Padarn

If you want to verify why no metrics sending out, how about using the built-in Slf4j reporter [1] which would record metrics in logs.

If you could view the metrics after enabled slf4j-reporter, you could then compare the configurations.

Best

Yun Tang

[1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter

---

From: Padarn Wilson <[hidden email]>
Sent: Thursday, May 30, 2019 18:20
To: user
Subject: [External] Flink 1.7.1 on EMR metrics

 

Hello all,

I am trying to run Flink 1.7.1 on EMR and having some trouble with metric reporting.

I was using the DataDogHttpReporter, but have also tried the StatsDReporter, but with both was seeing no metrics being collected.

To debug this I implemented my own reporter (based on StatsDReporter) and logged the name of the metric being sent:

private void send(final String name, final String value) {
   log.info("STATSD SENDING: ", name, value);
   try {
      String formatted = String.format("%s:%s|g", name, value);
      byte[] data = formatted.getBytes(StandardCharsets.UTF_8);
      socket.send(new DatagramPacket(data, data.length, this.address));
   }
   catch (IOException e) {
      LOG.error("unable to send packet to statsd at '{}:{}'", address.getHostName(), address.getPort());
   }
}

This code is certainly reached, because in my log I see a lot of this:

2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,351 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 
2019-05-30 10:18:40,352 INFO  com.grab.statsd.StatsDReporter                                - STATSD SENDING: 

As you can see, the name and value for the metric being reported is empty.

And the logs show everything initialized fine with no error:

2019-05-30 10:18:30,342 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Configuring stsd with {port=8125, host=127.0.0.1, class=com.grab.statsd.StatsDReporter}.
2019-05-30 10:18:30,344 INFO  com.grab.statsd.StatsDReporter                                - Configured StatsDReporter with {host:127.0.0.1, port:8125}
2019-05-30 10:18:30,344 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - Periodically reporting metrics in intervals of 10 SECONDS for reporter stsd of type com.grab.statsd.StatsDReporter.

Has anyone else  tried to work with Flink and metrics on EMR 1.7.1 (latest version on EMR). If so, any pointers as to what could be set up incorrectly?

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.