Question about the custom partitioner

[Xingcan Cui]

Hi all,

I want to duplicate records to multiple downstream tasks (not all of them thus the

Broadcasting should not work) in stream environment.

However, it seems that the current custom partitioner can return only one partition index.

Why this restriction exists or do I miss something?

Thanks,

Xingcan

[Aljoscha Krettek]

Hi,

I’m afraid that’s not possible out-of-box with the current APIs. I actually don’t know why the user-facing Partitioner only allows returning one target because the internal StreamPartitioner (which extends ChannelSelector) allows returning multiple target partitions.

You can hack around the API by manually creating your own StreamPartitioner and applying it to the DataStream as DataStream.partitionCustom() and DataStream.setConnectionType() (the first calls the latter) do.

Best,

Aljoscha

On 14. Jun 2017, at 09:09, Xingcan Cui <[hidden email]> wrote:

Hi all,

I want to duplicate records to multiple downstream tasks (not all of them thus the

Broadcasting should not work) in stream environment.

However, it seems that the current custom partitioner can return only one partition index.

Why this restriction exists or do I miss something?

Thanks,

Xingcan

[Xingcan Cui]

Hi Aljoscha,

Thanks for your explanation. I'll try what you suggests.

Best,

Xingcan

On Fri, Jun 16, 2017 at 5:19 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

I’m afraid that’s not possible out-of-box with the current APIs. I actually don’t know why the user-facing Partitioner only allows returning one target because the internal StreamPartitioner (which extends ChannelSelector) allows returning multiple target partitions.

You can hack around the API by manually creating your own StreamPartitioner and applying it to the DataStream as DataStream.partitionCustom() and DataStream.setConnectionType() (the first calls the latter) do.

Best,

Aljoscha

On 14. Jun 2017, at 09:09, Xingcan Cui <[hidden email]> wrote:

Hi all,

I want to duplicate records to multiple downstream tasks (not all of them thus the

Broadcasting should not work) in stream environment.

However, it seems that the current custom partitioner can return only one partition index.

Why this restriction exists or do I miss something?

Thanks,

Xingcan