Colocating Compute

Posted by Satyam Shekhar on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Colocating-Compute-tp37038.html

Hello,

I am using Flink v1.10 in a distributed environment to run SQL queries on batch and streaming data.

In my setup, data is sharded and distributed across the cluster. Each shard receives streaming updates from some external source. I wish to minimize data movement during query evaluation for performance reasons. For that, I need some construct to advise Flink planner to bind splits (shard) to the host where it is located. 

I have come across InputSplitAssigner which gives me levers to influence compute colocation for batch queries. Is there a way to do the same for streaming queries as well? 

Regards,

Satyam