Anomaly Detection with Flink-ML

[Branham, Jeremy [IT]]

Hello –

I’m working on an anomaly detector for some time series monitoring data.

I’ve setup an example project with Flink that reads from Kafka to get the monitoring data.

Unfortunately, I’m not sure what to do next.

 

The goal is to perform some clustering on the metric values that Flink is receiving and detect when the value[s] are anomalous.

I’ve got a DataStream<MetricDefinition>  that I think needs to go through some pre-processing, like transforming it into a vector, but I’m not sure how to proceed.

 

The pojo[MetricDefinition] looks like this –

 

https://github.com/savantly-net/metric-schema/blob/master/src/main/java/net/savantly/metrics/schema/MetricDefinition.java

 

Can anyone point me in the right direction?

 

Thanks!

 

 

Jeremy D. Branham

Technology Architect - Sprint
O: [phone] | M: [phone]

[hidden email]

#gettingbettereveryday

 

---

This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.

[Jonas Gröger]

Hello Jeremy,

it looks like what you are looking for is map (1 in, 1 out) / flatmap (1 in, 0-n out) for preprocessing on a single element basis as well as windows for looking at related MetricDefinition elements calculating some result.

I suggest you look into Windows (https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html) and basic transformations (https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/datastream_api.html#datastream-transformations).

EDIT: I must have missed the ML part of your question :) I have no idea about ML but my advice for Flink (the non-ML part) is still applicable :)

Regards,
Jonas

[Branham, Jeremy [IT]]

Great information!

Thanks Jonas!

------ Original message------

From: Jonas Gröger

Date: Fri, Jul 7, 2017 4:12 PM

To: [hidden email];

Cc:

Subject:Re: Anomaly Detection with Flink-ML

Hello Jeremy,

it looks like what you are looking for is map (1 in, 1 out) / flatmap (1 in,
0-n out) for preprocessing on a single element basis as well as windows for
looking at related MetricDefinition elements calculating some result.

I suggest you look into Windows
(https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.3%2Fdev%2Fwindows.html&data=02%7C01%7CJeremy.D.Branham%40sprint.com%7C66a475a8840f40aee51808d4c57cbe82%7C4f8bc0acbd784bf5b55f1b31301d9adf%7C0%7C0%7C636350587286223292&sdata=NoSCU2%2F3hcpLpwUCHdlFYrzA2dNraUuSMWlRUJ3H9vI%3D&reserved=0)
and basic transformations
(https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.3%2Fdev%2Fdatastream_api.html%23datastream-transformations&data=02%7C01%7CJeremy.D.Branham%40sprint.com%7C66a475a8840f40aee51808d4c57cbe82%7C4f8bc0acbd784bf5b55f1b31301d9adf%7C0%7C0%7C636350587286223292&sdata=KHnHRq9MSKTsRQJV4oUIAHJMHoxLlfaWWDCFqKsO95Y%3D&reserved=0).

Regards,
Jonas



--
View this message in context: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2FAnomaly-Detection-with-Flink-ML-tp14149p14151.html&data=02%7C01%7CJeremy.D.Branham%40sprint.com%7C66a475a8840f40aee51808d4c57cbe82%7C4f8bc0acbd784bf5b55f1b31301d9adf%7C0%7C0%7C636350587286223292&sdata=2PRn8Gb7vOAoCT7r91J4ewhTaVauSEXK380vZp4YV2Q%3D&reserved=0
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

---

This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.