placement preferences for streaming jobs

[StefaniaC]

Hi,

I have started using Flink and I am wondering if it is possible to specify placement preferences for the streaming jobs. More precisely, if I run Flink in stand-alone mode on a cluster and I submit a streaming job to it, is there a way to ask for the job or for some of its tasks to run on specific nodes?

Thank you in advance,
Stefania

[Stephan Ewen]

Hi Stefania!

I think there is no hook for that right now. If I understand you correctly, assuming you run YARN or so, you want to give the sources a set of hostnames, and when scheduling, the sources have preferences for those nodes.

Within a dataflow program (job), Flink will attempt to co-locate operations to minimize network traffic.

Greetings,

Stephan

On Fri, Nov 20, 2015 at 6:53 PM, Stefania Costache <[hidden email]> wrote:

Hi,

I have started using Flink and I am wondering if it is possible to specify placement preferences for the streaming jobs. More precisely, if I run Flink in stand-alone mode on a cluster and I submit a streaming job to it, is there a way to ask for the job or for some of its tasks to run on specific nodes?

Thank you in advance,
Stefania