Dealing with latency in Sink

[Mohit Anchlia]

What is the best way to dynamically adapt and tune down number of tasks created to write/read to a sink when sink slows down or the latency to sink increases? I am looking at the sink interface but don't see a way to influence flink to reduce the number of tasks or throttle the volume down to the sink. What is the best way to deal with this scenario?

[rmetzger0]

Hi Mohit,

Flink doesn't allow dynamic up or downscaling of parallel operator instances at runtime. However, you can stop and restore from a savepoint with a different parallelism. This way, you can adopt to workload changes.

Flink's handling of backpressure is very implicit. If you want to throttle a sink, you can just accept fewer records per second (by not returning from the invoke() call) to throttle the entire topology.

I think we can provide you much better support if you can provide us with some more details on your problem (which system do you want to connect to, what are your latency / throughput requirements, how much does the load change)

On Mon, Feb 6, 2017 at 10:34 PM, Mohit Anchlia <[hidden email]> wrote:

What is the best way to dynamically adapt and tune down number of tasks created to write/read to a sink when sink slows down or the latency to sink increases? I am looking at the sink interface but don't see a way to influence flink to reduce the number of tasks or throttle the volume down to the sink. What is the best way to deal with this scenario?