(DEPRECATED) Apache Flink User Mailing List archive.

Re:回复：How to get the evaluation result of a time-based window aggregation in time after a new event falling into the window?

classic Classic list List threaded Threaded
♦
♦
 2 messages Options
forideal
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re:回复：How to get the evaluation result of a time-based window aggregation in time after a new event falling into the window?

forideal
Hi Chengcheng Zhang,

      You are welcome.
      I also got the material from the community to answer your question.
      There is also a hidden method here, as if the community has not been placed publicly in the document.
      table.exec.emit.early-fire.enabled = true
table.exec.emit.early-fire.delay = 60 s

Best, forideal






At 2020-08-16 13:21:25, "Chengcheng Zhang" <[hidden email]> wrote:

Hi, forideal
Thank you so much, it does help a lot.
The approach you mentioned earlier, happened to be the same path we took two days ago, and it worked well as expected. 
To be honest, after some effort-taking searches on the Internet, I' am a little convinced that, this maybe the best solution at the moment. However, the time-based window aggregation is a great feature in Flink, as we all know. Would it be perfect if we could use time-based windows and still get the latest result at the same time?
Best, Chengcheng Zhang


------------------ 原始邮件 ------------------
发件人: "forideal" <[hidden email]>;
发送时间: 2020年8月16日(星期天) 中午12:24
收件人: "Chengcheng Zhang"<[hidden email]>;
抄送: "user"<[hidden email]>;
主题: Re:How to get the evaluation result of a time-based window aggregation in time after a new event falling into the window?

Hi Chengcheng Zhang,
 
          Is this your scene? For example, every day is divided into 12 hours, let’s take today as an example, 2020081600 2020081601,...2020081623
For example, if we count pv, we can count like this
INSERT INTO cumulative_pv
SELECT time_str, count(1)
FROM pv_per_hour
GROUP BY time_str;
In this sql, time_str is an hour in 2020081600, 2020081601,...2020081623.


Hope this helps.

Best, forideal



At 2020-08-16 12:05:04, "Chengcheng Zhang" <[hidden email]> wrote:

Hi,
I'm a new user of Flink, and have been puzzled a lot by the time-based window aggregation result.
For our business, hourly and daily reports have to been created best in a real time style.  So, I used a event-time based window aggregation to consume the Kafka  data stream, but found that, only after the current hour or day passed, the newest result could be seen on console or upserted to MySQL.
How can I get the latest window result immediately after a stream record falling into it? Is there a specific configuration option for this, hopefully? Please help and rescue me.
Best regards.


 



 
Theo
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: 回复：How to get the evaluation result of a time-based window aggregation in time after a new event falling into the window?

Theo
Hi Chengcheng Zhang,

I think your request is related to this feature request from two years ago here [1], with me asking about the status one year ago [2].
You might want to upvote this so we can hope that it gets some more attention in future.

Today, it is possible to write your own DataStream API where you customize the triggers to your wishes (CountTigger of 1), but with Flink SQL, you sadly lose most of that flexibility.

Thanks @forideal  for mentioning that there is something in the configuration. I wasn't aware of that.

Best regards
Theo

Von: "forideal" <[hidden email]>
An: "Chengcheng Zhang" <[hidden email]>
CC: "user" <[hidden email]>
Gesendet: Sonntag, 16. August 2020 09:14:23
Betreff: Re:回复：How to get the evaluation result of a time-based window aggregation in time after a new event falling into the window?

Hi Chengcheng Zhang,

      You are welcome.
      I also got the material from the community to answer your question.
      There is also a hidden method here, as if the community has not been placed publicly in the document.
      table.exec.emit.early-fire.enabled = true
table.exec.emit.early-fire.delay = 60 s

Best, forideal






At 2020-08-16 13:21:25, "Chengcheng Zhang" <[hidden email]> wrote:

Hi, forideal
Thank you so much, it does help a lot.
The approach you mentioned earlier, happened to be the same path we took two days ago, and it worked well as expected. 
To be honest, after some effort-taking searches on the Internet, I' am a little convinced that, this maybe the best solution at the moment. However, the time-based window aggregation is a great feature in Flink, as we all know. Would it be perfect if we could use time-based windows and still get the latest result at the same time?
Best, Chengcheng Zhang


------------------ 原始邮件 ------------------
发件人: "forideal" <[hidden email]>;
发送时间: 2020年8月16日(星期天) 中午12:24
收件人: "Chengcheng Zhang"<[hidden email]>;
抄送: "user"<[hidden email]>;
主题: Re:How to get the evaluation result of a time-based window aggregation in time after a new event falling into the window?

Hi Chengcheng Zhang,
 
          Is this your scene? For example, every day is divided into 12 hours, let’s take today as an example, 2020081600 2020081601,...2020081623
For example, if we count pv, we can count like this
INSERT INTO cumulative_pv
SELECT time_str, count(1)
FROM pv_per_hour
GROUP BY time_str;
In this sql, time_str is an hour in 2020081600, 2020081601,...2020081623.


Hope this helps.

Best, forideal



At 2020-08-16 12:05:04, "Chengcheng Zhang" <[hidden email]> wrote:

Hi,
I'm a new user of Flink, and have been puzzled a lot by the time-based window aggregation result.
For our business, hourly and daily reports have to been created best in a real time style.  So, I used a event-time based window aggregation to consume the Kafka  data stream, but found that, only after the current hour or day passed, the newest result could be seen on console or upserted to MySQL.
How can I get the latest window result immediately after a stream record falling into it? Is there a specific configuration option for this, hopefully? Please help and rescue me.
Best regards.


 
Free forum by Nabble Edit this page