Flink suggestions;

[Aissa Elaffani]

Hello Guys,

I am a beginner in this field of real-time streaming and i am working with apache flink, and i ignore a lot of features of it, and actually I am building an application, in which i receive some sensors data in this format {"status": "Alerte", "classe": " ", "value": {"temperature": 15.7}, "mode": "ON", "equipementID": 1, "date": "2019-03-20 22:00", "sensorID": 9157}, each sensor is installed on an equipment in a workshop in a factory somewhere. My goal is :

If one sensor of a factory get down (status="alerte"), I want that the status of all the factory to be Alerte. But as  the Stream does not contain the factory ID, other Static data set source that contain the data of factories and the sensors that belongs to each one.

So Please guys i want to know the optimized way to do so, and the aggregation that i need to do!

Sorry for disturbing you, i wish you all the best! And i hope you share with me the of your experiences!

Best regards,

Aissa

[Chesnay Schepler]

Am I understanding you correctly in that, if one sensor of on factory
raises an alert, then you want all sensors in that same factory to raise
an alert?

How big is this dataset that maps sensors to factories?

Maybe you can just load them into a Map<Sensor, Factory> in say a
FlatMap, enrich the sensor data stream, keyBy the factory id and then do
your windowing logic (I assume you have some time window)?
Although this would only work if you want alerts only for sensors in a
given window.

If you want to have an alert for literally all sensors in a factory then
you could instead add the FlatMap after the window, and map the sensor
-> factory -> all_sensors and duplicate the alerts.

On 15/05/2020 08:21, Aissa Elaffani wrote:

> Hello Guys,
> I am a beginner in this field of real-time streaming and i am working
> with apache flink, and i ignore a lot of features of it, and actually
> I am building an application, in which i receive some sensors data in
> this format {"status": "Alerte", "classe": " ", "value":
> {"temperature": 15.7}, "mode": "ON", "equipementID": 1, "date":
> "2019-03-20 22:00", "sensorID": 9157}, each sensor is installed on an
> equipment in a workshop in a factory somewhere. My goal is :
> If one sensor of a factory get down (status="alerte"), I want that the
> status of all the factory to be Alerte. But as the Stream does not
> contain the factory ID, other Static data set source that contain the
> data of factories and the sensors that belongs to each one.
> So Please guys i want to know the optimized way to do so, and the
> aggregation that i need to do!
> Sorry for disturbing you, i wish you all the best! And i hope you
> share with me the of your experiences!
> Best regards,
> Aissa

[Arvid Heise-3]

Hi Aissa,

Your use case is quite unusual. You usually have an alert in a dashboard (potentially sending an email), if any of the sensors show an error.  You usually want to retain the original error code to be able to quickly identify the issue.

Your use case would make sense if you want to filter the data for all sensors. But usually, you do it downstream when you aggregate the events.

For example, you want to determine the average temperature; at that point, you could also aggregate the #failed sensors. Then you can filter on the #failed field.

Another common pattern is to just ignore the value of the failed sensor in aggregation. That's achieved by just filtering the record initially.

On Fri, May 15, 2020 at 10:31 AM Chesnay Schepler <[hidden email]> wrote:

Am I understanding you correctly in that, if one sensor of on factory
raises an alert, then you want all sensors in that same factory to raise
an alert?

How big is this dataset that maps sensors to factories?

Maybe you can just load them into a Map<Sensor, Factory> in say a
FlatMap, enrich the sensor data stream, keyBy the factory id and then do
your windowing logic (I assume you have some time window)?
Although this would only work if you want alerts only for sensors in a
given window.

If you want to have an alert for literally all sensors in a factory then
you could instead add the FlatMap after the window, and map the sensor
-> factory -> all_sensors and duplicate the alerts.

On 15/05/2020 08:21, Aissa Elaffani wrote:
> Hello Guys,
> I am a beginner in this field of real-time streaming and i am working
> with apache flink, and i ignore a lot of features of it, and actually
> I am building an application, in which i receive some sensors data in
> this format {"status": "Alerte", "classe": " ", "value":
> {"temperature": 15.7}, "mode": "ON", "equipementID": 1, "date":
> "2019-03-20 22:00", "sensorID": 9157}, each sensor is installed on an
> equipment in a workshop in a factory somewhere. My goal is :
> If one sensor of a factory get down (status="alerte"), I want that the
> status of all the factory to be Alerte. But as the Stream does not
> contain the factory ID, other Static data set source that contain the
> data of factories and the sensors that belongs to each one.
> So Please guys i want to know the optimized way to do so, and the
> aggregation that i need to do!
> Sorry for disturbing you, i wish you all the best! And i hope you
> share with me the of your experiences!
> Best regards,
> Aissa

--