Use Partitioner to forward messages to subtask by index

[Joshua Griffith-2]

Is it possible to use a custom Partitioner to forward messages to a downstream substask by the subtask's index? I realize that it would not be safe to rely upon this across job restarts but this does not effect my particular application. I attempted to return a partition number identical to the subtask number but this did not work. For example, assigning a message to partition 1 would send it to subtask 0.

Thanks,

Josh

[Ken Krugler]

Hi Joshua,

It is possible, but fragile, as it depends on the internal code that Flink uses to calculate a hash for a key, and the max operator parallelism, etc.

See makeKeyForOperatorIndex, which will generate a String that can be used for partitioning to a specific subtask (operator index)

— Ken

On Jun 21, 2019, at 10:15 AM, Joshua Griffith <[hidden email]> wrote:

Is it possible to use a custom Partitioner to forward messages to a downstream substask by the subtask's index? I realize that it would not be safe to rely upon this across job restarts but this does not effect my particular application. I attempted to return a partition number identical to the subtask number but this did not work. For example, assigning a message to partition 1 would send it to subtask 0.

Thanks,

Josh

--------------------------

Ken Krugler
[phone]
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra