Partition index from partitionCustom vs getIndexOfThisSubtask downstream

[Urs Schoenenberger]

Hi,

if I use DataStream::partitionCustom, will the partition number that my
custom Partitioner<K> returns always be equal to getIndexOfThisSubtask
in the following operator?

A test case with different parallelisms seems to suggest this is true,
but the Javadoc seems ambiguous to me since the Partitioner doc talks
about the "partition index" while the RuntimeContext doc talks about the
"index of the parallel subtask".

Thanks,
Urs

--
Urs Schönenberger - [hidden email]

TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082

[Tzu-Li (Gordon) Tai]

Hi Urs,

Yes, the returned “index” from the custom partitioner refers to the parallel subtask index.

I agree that the mismatching terminology used could be slightly misleading. Could you open a JIRA to improve the Javadoc for that? Thanks!

Cheers,

Gordon

On 27 June 2017 at 10:40:47 PM, Urs Schoenenberger ([hidden email]) wrote:

Hi,

if I use DataStream::partitionCustom, will the partition number that my
custom Partitioner<K> returns always be equal to getIndexOfThisSubtask
in the following operator?

A test case with different parallelisms seems to suggest this is true,
but the Javadoc seems ambiguous to me since the Partitioner doc talks
about the "partition index" while the RuntimeContext doc talks about the
"index of the parallel subtask".

Thanks,
Urs

--
Urs Schönenberger - [hidden email]

TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082