Custom time window in Flink
Locked
Custom time window in Flink
 
|
Hello,
I wanted to enquire how a job I am trying to do with Flink can be done. I have also posted a question on StackOverflow. PFB the link: I am using Flink's
TimeWindow functionality to perform some computations. I am creating a 5 minute Window. However I want to create a one hour Window for only the first time. The next Windows I need are of 5 minutes. Such that for the first hour, data is collected and my operation is
performed on it. Once this is done, every five minutes the same
operation is performed. Can you kindly help me with this? How can such a functionality be implemented? Thanks and Regards,  |
RE: Custom time window in Flink
|
|
Hi, Tha way to do this is to create your own evictor. In the evictor you can than decide when the events are removed. I would suggest creating a symmetric trigger
as well because I would assume that you also need to fire the computation first after 1 hour and then at each 5 minutes. The logic would be that you have a field that marks whether a window was created or not (e.g., a Boolean field in the evictor class), which once a window is
created you can set it to false and from there one you operate on 5 minutes windows Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail:
[hidden email] Mobile: +49 15209084330 Telephone: +49 891588344173
HUAWEI TECHNOLOGIES Duesseldorf GmbH This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use
of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the
sender by phone or email immediately and delete it! From: Piyush Shrivastava [mailto:[hidden email]]
Hello, I wanted to enquire how a job I am trying to do with Flink can be done. I have also posted a question on StackOverflow. PFB the link: I am using Flink's
Such that for the first hour, data is collected and my operation is performed on it. Once this is done, every five minutes the same
operation is performed. Can you kindly help me with this? How can such a functionality be implemented? Thanks and Regards,
|
Re: Custom time window in Flink
|
|
Hello, Thanks a lot for your reply. Can you share a sample code or example which I can refer while creating a custom evictor and trigger? On Wednesday, 20 April 2016 2:50 PM, Radu Tudoran <[hidden email]> wrote: Hi,
Tha way to do this is to create your own evictor. In the evictor you can than decide when the events are removed. I would suggest creating a symmetric trigger
as well because I would assume that you also need to fire the computation first after 1 hour and then at each 5 minutes.
The logic would be that you have a field that marks whether a window was created or not (e.g., a Boolean field in the evictor class), which once a window is
created you can set it to false and from there one you operate on 5 minutes windows
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
 HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail:
[hidden email]
Mobile: +49 15209084330
Telephone: +49 891588344173
HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063, Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063, Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use
of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the
sender by phone or email immediately and delete it!
From: Piyush Shrivastava [mailto:[hidden email]]
Sent: Wednesday, April 20, 2016 9:59 AM To: [hidden email] Subject: Custom time window in Flink Hello,
I wanted to enquire how a job I am trying to do with Flink can be done.
I have also posted a question on StackOverflow. PFB the link:
I am using Flink's
TimeWindow functionality to perform some computations. I am creating
a 5 minute Window. However I want to create a one hour
Window for only the first time. The next Windows I need are of 5
minutes. Such that for the first hour, data is collected and my operation is performed on it. Once this is done, every five minutes the same
operation is performed.
Can you kindly help me with this? How can such a functionality be implemented?
Thanks and Regards,
|
RE: Custom time window in Flink
|
|
Hi, Easiest way is to just start from the code of an existing one If you take the example of Timeevictor you would just need to use the same code and modify the public int evict class. Same story with the triggers https://github.com/apache/flink/tree/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail:
[hidden email] Mobile: +49 15209084330 Telephone: +49 891588344173
HUAWEI TECHNOLOGIES Duesseldorf GmbH This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use
of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the
sender by phone or email immediately and delete it! From: Piyush Shrivastava [mailto:[hidden email]]
Hello, Thanks a lot for your reply. Can you share a sample code or example which I can refer while creating a custom evictor and trigger? On Wednesday, 20 April 2016 2:50 PM, Radu Tudoran <[hidden email]> wrote: Hi, Tha way to do this is to create your own evictor. In the evictor you can than decide when the events are removed. I would suggest creating
a symmetric trigger as well because I would assume that you also need to fire the computation first after 1 hour and then at each 5 minutes. The logic would be that you have a field that marks whether a window was created or not (e.g., a Boolean field in the evictor class),
which once a window is created you can set it to false and from there one you operate on 5 minutes windows Dr. Radu Tudoran Research Engineer - Big Data Expert IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail:
[hidden email] Mobile: +49 15209084330 Telephone: +49 891588344173 HUAWEI TECHNOLOGIES Duesseldorf GmbH This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose
address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this
e-mail in error, please notify the sender by phone or email immediately and delete it! From: Piyush Shrivastava [[hidden email]]
Hello, I wanted to enquire how a job I am trying to do with Flink can be done. I have also posted a question on StackOverflow. PFB the link: I am using Flink's
Such that for the first hour, data is collected and my operation is performed on it. Once this is done, every five minutes the same
operation is performed. Can you kindly help me with this? How can such a functionality be implemented? Thanks and Regards,
|
| Free forum by Nabble | Edit this page |