hourly counter

[Lian Jiang]

Hi,

I have a window function with a window width of 1 min. I want to have an hourly counter which is reset every hour so it never overflows. There are multiple ways but none of them is straightforward:

StatsDClient instance = new NonBlockingStatsDClientBuilder()

int count = 0;

void incr() {
  metricClient.count("mycounter", 1, "mytag");

  count++;

}

void reset() {
   metricClient.count("mycounter", -count, "mytag");

   count = 0;

}

As you can see, the code needs to maintain a "count" variable to reset mycounter. 
Also since timer is not available in Window function, extra code is needed to reset mycounter every hour. 
Is there an easier way for implementing hourly counter? Or it is not a concern that a counter will overflow?

Thanks

Lian

[Timo Walther]

Hi Lian,

you are right that timers are not available in a ProcessWindowFunction
but the state store can be accessed. So given that your window width is
1 min, you could maintain an additional state value for counting the
minutes and updating your counter once this value reached 60.

Otherwise, I would recommend to use a process function and implement the
windowing logic yourself if it is a simple windowing operation.

Btw if you are using a Long counter, I would say that chances are low
that it will overflow. Also, have you considered using Flink's metric
system? it might make custom metric clients unnecessary.

I hope this helps.

Regards,
Timo


On 22.09.20 02:02, Lian Jiang wrote:

> Hi,
>
> I have a window function with a window width of 1 min. I want to have an
> hourly counter which is reset every hour so it never overflows. There
> are multiple ways but none of them is straightforward:
>
> StatsDClient instance =new NonBlockingStatsDClientBuilder()
>
> int count = 0;
>
> void incr() {
> metricClient.count("mycounter",1,"mytag");
>
>    count++;
>
> }
>
> void reset() {
> metricClient.count("mycounter",-count,"mytag");
>
>     count = 0;
>
> }
>
> As you can see, the code needs to maintain a "count" variable to reset mycounter.
> Also since timer is not available in Window function, extra code is needed to reset mycounter every hour.
> Is there an easier way for implementing hourly counter? Or it is not a concern that a counter will overflow?
>
> Thanks
>
> Lian
>
>
>