Re: [EXTERNAL] Re: Delayed processing and Rate limiting

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: Delayed processing and Rate limiting

PoolakkalMukkath, Shakir

Hi Victor,

 

Thanks for the reply and it helps. For the delayed processing, this is exactly what I was looking for.

 

But for the Rate Limit, the one you suggested can only control the number of parallel requests. What I am looking is like limit the number of request per second or minute etc. Like 100 requests per minute, anything more that should trigger backpressure.

A reference to how we can create the backpressure based on condition will also help me if I need to build my own rate limiter.

 

Thanks in advance

Shakir

 

From: Victor Wong <[hidden email]>
Date: Wednesday, August 7, 2019 at 10:59 AM
To: "PoolakkalMukkath, Shakir" <[hidden email]>, user <[hidden email]>
Subject: [EXTERNAL] Re: Delayed processing and Rate limiting

 

Hi Shakir,

 

> Delayed Processing

Maybe you can make use of the function ‘org.apache.flink.streaming.api.TimerService#registerProcessingTimeTimer’, check this doc for more details:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#example

 

> Rate Limit External Data Access

With AsyncFunction, you can set a ‘capacity’ which defines how many asynchronous requests may be in progress at the same time, I’m not sure if this is what you need or not.

Check this doc for more details: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html#async-io-api

 

Best,

Victor

 

From: "PoolakkalMukkath, Shakir" <[hidden email]>
Date: Wednesday, August 7, 2019 at 10:06 PM
To: user <[hidden email]>
Subject: Delayed processing and Rate limiting

 

Hi Flink Team,

 

I am looking for some direction/recommendation for below tasks

 

  1. Delayed Processing:

Having a use case where we need to process events after a time-delay from event time. Let’s say, the event happened at time t1, and it reached the Flink immediately, but I have to wait t1+2min to process this.

We are sourcing the events from Kafka, we like this applied after SourceFn. May be we can do this by ThreadSleep() on a MapFn, but looking if there is a better way to achieve this.  

 

  1. Rate Limit External Data Access

The best practices to implement rate limiting to external service, it can be either on mapFn or AsynchFn. What is the recommended approach to rate limit and  build backpressure.

 

Thanks in advance

 

Thanks,

Shakir

Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: Delayed processing and Rate limiting

Biao Liu
Hi Shakir,

I'm not sure I have fully understand your requirements. I'll try to answer your questions. 

From my understanding, there is no built-in feature of Flink to support "rate limit" directly. I guess you need to implement one yourself.
Both of MapFunction or AsyncFunction could satisfy your requirement well IMO. The difference between them is whether you need an asynchronous processing or not. [1]

Currently back pressure of Flink could not base on user-defined conditions. It's an internal flow-control strategy of Flink. [2] I don't think that's the thing you want.


On Thu, Aug 8, 2019 at 4:13 AM PoolakkalMukkath, Shakir <[hidden email]> wrote:

Hi Victor,

 

Thanks for the reply and it helps. For the delayed processing, this is exactly what I was looking for.

 

But for the Rate Limit, the one you suggested can only control the number of parallel requests. What I am looking is like limit the number of request per second or minute etc. Like 100 requests per minute, anything more that should trigger backpressure.

A reference to how we can create the backpressure based on condition will also help me if I need to build my own rate limiter.

 

Thanks in advance

Shakir

 

From: Victor Wong <[hidden email]>
Date: Wednesday, August 7, 2019 at 10:59 AM
To: "PoolakkalMukkath, Shakir" <[hidden email]>, user <[hidden email]>
Subject: [EXTERNAL] Re: Delayed processing and Rate limiting

 

Hi Shakir,

 

> Delayed Processing

Maybe you can make use of the function ‘org.apache.flink.streaming.api.TimerService#registerProcessingTimeTimer’, check this doc for more details:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#example

 

> Rate Limit External Data Access

With AsyncFunction, you can set a ‘capacity’ which defines how many asynchronous requests may be in progress at the same time, I’m not sure if this is what you need or not.

Check this doc for more details: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html#async-io-api

 

Best,

Victor

 

From: "PoolakkalMukkath, Shakir" <[hidden email]>
Date: Wednesday, August 7, 2019 at 10:06 PM
To: user <[hidden email]>
Subject: Delayed processing and Rate limiting

 

Hi Flink Team,

 

I am looking for some direction/recommendation for below tasks

 

  1. Delayed Processing:

Having a use case where we need to process events after a time-delay from event time. Let’s say, the event happened at time t1, and it reached the Flink immediately, but I have to wait t1+2min to process this.

We are sourcing the events from Kafka, we like this applied after SourceFn. May be we can do this by ThreadSleep() on a MapFn, but looking if there is a better way to achieve this.  

 

  1. Rate Limit External Data Access

The best practices to implement rate limiting to external service, it can be either on mapFn or AsynchFn. What is the recommended approach to rate limit and  build backpressure.

 

Thanks in advance

 

Thanks,

Shakir