Calculating metrics real time

[Rahul Raj]

Hi,

I have to calculate some complicated metrics like click through rate , click value rate and conversions on real time data using flink. But I am not sure what functionality of flink should I use to program this because it involves collection of some records in memory for certain time may be 1 minute and then using formula for calculating metrics on those records. 

Am I correct with my approach or is there any preferred approach for such tasks in flink? Can I use windows for doing this? Any tutorial or example would be really great.

Rahul Raj

[Fabian Hueske-2]

Hi,

I'd suggest to have a look at the window operators [1]. For example a tumbling window of 1 minute can be used to compute metrics every minute.

Flink's window operators are very extensible and you can implement very custom logic if the predefined windows don't match your use case. In any case, windows are used to collect and perform a computation on a set of records.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html

2017-10-05 12:57 GMT+02:00 Rahul Raj <[hidden email]>:

Hi,

I have to calculate some complicated metrics like click through rate , click value rate and conversions on real time data using flink. But I am not sure what functionality of flink should I use to program this because it involves collection of some records in memory for certain time may be 1 minute and then using formula for calculating metrics on those records. 

Am I correct with my approach or is there any preferred approach for such tasks in flink? Can I use windows for doing this? Any tutorial or example would be really great.

Rahul Raj

[Rahul Raj]

Thanks for the response. So, i am guessing windows in flink will store the records in memory before processing them. Correct?

Rahul Raj

On Oct 5, 2017 17:50, "Fabian Hueske" <[hidden email]> wrote:

Hi,

I'd suggest to have a look at the window operators [1]. For example a tumbling window of 1 minute can be used to compute metrics every minute.

Flink's window operators are very extensible and you can implement very custom logic if the predefined windows don't match your use case. In any case, windows are used to collect and perform a computation on a set of records.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html

2017-10-05 12:57 GMT+02:00 Rahul Raj <[hidden email]>:

Hi,

I have to calculate some complicated metrics like click through rate , click value rate and conversions on real time data using flink. But I am not sure what functionality of flink should I use to program this because it involves collection of some records in memory for certain time may be 1 minute and then using formula for calculating metrics on those records. 

Am I correct with my approach or is there any preferred approach for such tasks in flink? Can I use windows for doing this? Any tutorial or example would be really great.

Rahul Raj

[Fabian Hueske-2]

That depends on the state backend [1] that you are using.

If you use the RocksDBStateBackend, state is written to RocksDB which persists to disk.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/ops/state_backends.html

2017-10-05 14:41 GMT+02:00 Rahul Raj <[hidden email]>:

Thanks for the response. So, i am guessing windows in flink will store the records in memory before processing them. Correct?

Rahul Raj

On Oct 5, 2017 17:50, "Fabian Hueske" <[hidden email]> wrote:

Hi,

I'd suggest to have a look at the window operators [1]. For example a tumbling window of 1 minute can be used to compute metrics every minute.

Flink's window operators are very extensible and you can implement very custom logic if the predefined windows don't match your use case. In any case, windows are used to collect and perform a computation on a set of records.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html

2017-10-05 12:57 GMT+02:00 Rahul Raj <[hidden email]>:

Hi,

I have to calculate some complicated metrics like click through rate , click value rate and conversions on real time data using flink. But I am not sure what functionality of flink should I use to program this because it involves collection of some records in memory for certain time may be 1 minute and then using formula for calculating metrics on those records. 

Am I correct with my approach or is there any preferred approach for such tasks in flink? Can I use windows for doing this? Any tutorial or example would be really great.

Rahul Raj