user driven stream processing

[zanqing zhang]

Hi All,

Has anyone done any stream processing driven by a user request? What's the recommended way of doing this? Or is this completely wrong direction to go for applications running on top of Flink?

Basically we need to tweak the stream processing based on parameters provided by a user, e.g. show me the total # of application failures due to "ABC", which is provided by the user. We are thinking of starting a flink job with "ABC" as a parameter but this would result in a huge number of flink jobs, is there a better way for this? Can we trigger the calculation on a running job?

Thanks in advance. 

KZ

[Tony Wei]

Hi KZ,

https://data-artisans.com/blog/real-time-fraud-detection-ing-bank-apache-flink

This article seems to be a good example to trigger a new calculation on a running job. Maybe you can get some help from it.

Best Regards,

Tony Wei

2017-11-29 4:53 GMT+08:00 zanqing zhang <[hidden email]>:

Hi All,

Has anyone done any stream processing driven by a user request? What's the recommended way of doing this? Or is this completely wrong direction to go for applications running on top of Flink?

Basically we need to tweak the stream processing based on parameters provided by a user, e.g. show me the total # of application failures due to "ABC", which is provided by the user. We are thinking of starting a flink job with "ABC" as a parameter but this would result in a huge number of flink jobs, is there a better way for this? Can we trigger the calculation on a running job?

Thanks in advance. 

KZ

[Fabian Hueske-2]

Another example is King's RBEA platform [1] which was built on Flink.

In a nutshell, RBEA runs a single large Flink job, to which users can add queries that should be computed.
Of course, the query language is restricted because they queries must match on the structure of the running job.

Hope this helps,

Fabian

[1] http://2016.flink-forward.org/kb_sessions/rbea-scalable-real-time-analytics-at-king/

2017-11-29 3:32 GMT+01:00 Tony Wei <[hidden email]>:

Hi KZ,

https://data-artisans.com/blog/real-time-fraud-detection-ing-bank-apache-flink

This article seems to be a good example to trigger a new calculation on a running job. Maybe you can get some help from it.

Best Regards,

Tony Wei

2017-11-29 4:53 GMT+08:00 zanqing zhang <[hidden email]>:

Hi All,

Has anyone done any stream processing driven by a user request? What's the recommended way of doing this? Or is this completely wrong direction to go for applications running on top of Flink?

Basically we need to tweak the stream processing based on parameters provided by a user, e.g. show me the total # of application failures due to "ABC", which is provided by the user. We are thinking of starting a flink job with "ABC" as a parameter but this would result in a huge number of flink jobs, is there a better way for this? Can we trigger the calculation on a running job?

Thanks in advance. 

KZ

[zanqing zhang]

Thanks Fabian and Tony for the info. It's very helpful.

Looks like the general approach is to implement a job topology containing parameterized (CoXXXMapFunction) operators. The user defined parameters will be ingested using the extra input the CoXXXMapFunction take. 

Ken

On Wed, Nov 29, 2017 at 5:27 AM, Fabian Hueske <[hidden email]> wrote:

Another example is King's RBEA platform [1] which was built on Flink.

In a nutshell, RBEA runs a single large Flink job, to which users can add queries that should be computed.
Of course, the query language is restricted because they queries must match on the structure of the running job.

Hope this helps,

Fabian

[1] http://2016.flink-forward.org/kb_sessions/rbea-scalable-real-time-analytics-at-king/

2017-11-29 3:32 GMT+01:00 Tony Wei <[hidden email]>:

Hi KZ,

https://data-artisans.com/blog/real-time-fraud-detection-ing-bank-apache-flink

This article seems to be a good example to trigger a new calculation on a running job. Maybe you can get some help from it.

Best Regards,

Tony Wei

2017-11-29 4:53 GMT+08:00 zanqing zhang <[hidden email]>:

Hi All,

Has anyone done any stream processing driven by a user request? What's the recommended way of doing this? Or is this completely wrong direction to go for applications running on top of Flink?

Basically we need to tweak the stream processing based on parameters provided by a user, e.g. show me the total # of application failures due to "ABC", which is provided by the user. We are thinking of starting a flink job with "ABC" as a parameter but this would result in a huge number of flink jobs, is there a better way for this? Can we trigger the calculation on a running job?

Thanks in advance. 

KZ

--