throttled stream

[Chen Bekor]

is there a way to consume a kafka stream using flink with  a predefined rate limit (eg 5 events per second)

we need this because we need to control some 3rd party api rate limitations so,  even if we have a much larger throughput potential, we must control the consumption rate in order not to overflow the API channel.

[Niels Basjes-2]

Simple idea: create a map function that only does "sleep 1/5 second" and put that in your pipeline somewhere.

Niels

On 16 Apr 2016 22:38, "Chen Bekor" <[hidden email]> wrote:

is there a way to consume a kafka stream using flink with  a predefined rate limit (eg 5 events per second)

we need this because we need to control some 3rd party api rate limitations so,  even if we have a much larger throughput potential, we must control the consumption rate in order not to overflow the API channel.

[Márton Balassi]

There is a utility in flink-streaming-examples that might be useful, but is generally the same idea that Niels suggests. [1]

[1] https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java

On Sun, Apr 17, 2016 at 8:42 AM, Niels Basjes <[hidden email]> wrote:

Simple idea: create a map function that only does "sleep 1/5 second" and put that in your pipeline somewhere.

Niels

On 16 Apr 2016 22:38, "Chen Bekor" <[hidden email]> wrote:

is there a way to consume a kafka stream using flink with  a predefined rate limit (eg 5 events per second)

we need this because we need to control some 3rd party api rate limitations so,  even if we have a much larger throughput potential, we must control the consumption rate in order not to overflow the API channel.

[rmetzger0]

Hi,

I would also go for Niels approach. If the mapper has the same parallelism as the source and its right after it, it'll be chained to the source. The throttling then happens with almost no overhead.

Regarding the ThrottledIterator: Afaik there is no iterator involved when reading data out of the Kafka connector, so there is no way to plug it in anywhere.

On Sun, Apr 17, 2016 at 8:53 AM, Márton Balassi <[hidden email]> wrote:

There is a utility in flink-streaming-examples that might be useful, but is generally the same idea that Niels suggests. [1]

[1] https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java

On Sun, Apr 17, 2016 at 8:42 AM, Niels Basjes <[hidden email]> wrote:

Simple idea: create a map function that only does "sleep 1/5 second" and put that in your pipeline somewhere.

Niels

On 16 Apr 2016 22:38, "Chen Bekor" <[hidden email]> wrote:

is there a way to consume a kafka stream using flink with  a predefined rate limit (eg 5 events per second)

we need this because we need to control some 3rd party api rate limitations so,  even if we have a much larger throughput potential, we must control the consumption rate in order not to overflow the API channel.