DataSetUtils zipWithIndex question

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

DataSetUtils zipWithIndex question

Tarandeep Singh
It works in two phases/steps
1) Count number of elements in each partition (using mapPartition)
2) In second mapPartition, unique ID is assigned by calculating offset using number of elements computed in step 1.

Is there any chance the second mapPartition won't get same number of elements as first mapPartition (assuming data is in HDFS)?

Thanks
Tarandeep
Reply | Threaded
Open this post in threaded view
|

Re: DataSetUtils zipWithIndex question

Till Rohrmann
Hi Tarandeep,

the number of elements in each partition should stay constant. In fact the elements in each partition should not change.

Cheers,
Till

On Wed, Mar 30, 2016 at 8:14 AM, Tarandeep Singh <[hidden email]> wrote:
It works in two phases/steps
1) Count number of elements in each partition (using mapPartition)
2) In second mapPartition, unique ID is assigned by calculating offset using number of elements computed in step 1.

Is there any chance the second mapPartition won't get same number of elements as first mapPartition (assuming data is in HDFS)?

Thanks
Tarandeep

Reply | Threaded
Open this post in threaded view
|

Re: DataSetUtils zipWithIndex question

Flavio Pompermaier
Hi Till and Tarandeep,
I'm also interested in better understanding my knowledge about the concept of a partition..
From what I know a partition is the portion of data assigned by the job manager to each task manager..right?
Then, each partition is divided again at the task manager to maximize the slot usage..is it correct?
In every case, there will be a case where at least one partition is smaller than the others...am I wrong? Am I confusing some term..?

Best,
Flavio


On Thu, Mar 31, 2016 at 1:56 PM, Till Rohrmann <[hidden email]> wrote:
Hi Tarandeep,

the number of elements in each partition should stay constant. In fact the elements in each partition should not change.

Cheers,
Till

On Wed, Mar 30, 2016 at 8:14 AM, Tarandeep Singh <[hidden email]> wrote:
It works in two phases/steps
1) Count number of elements in each partition (using mapPartition)
2) In second mapPartition, unique ID is assigned by calculating offset using number of elements computed in step 1.

Is there any chance the second mapPartition won't get same number of elements as first mapPartition (assuming data is in HDFS)?

Thanks
Tarandeep


Reply | Threaded
Open this post in threaded view
|

Re: DataSetUtils zipWithIndex question

Till Rohrmann
A partition is the portion of data each task receives. Thus, the degree of parallelism of your program/task decides how many different partitions you have. Depending on the upstream operators (and which data is send to which task), the partitions will most likely differ in size.

Cheers,
Till

On Thu, Mar 31, 2016 at 2:11 PM, Flavio Pompermaier <[hidden email]> wrote:
Hi Till and Tarandeep,
I'm also interested in better understanding my knowledge about the concept of a partition..
From what I know a partition is the portion of data assigned by the job manager to each task manager..right?
Then, each partition is divided again at the task manager to maximize the slot usage..is it correct?
In every case, there will be a case where at least one partition is smaller than the others...am I wrong? Am I confusing some term..?

Best,
Flavio


On Thu, Mar 31, 2016 at 1:56 PM, Till Rohrmann <[hidden email]> wrote:
Hi Tarandeep,

the number of elements in each partition should stay constant. In fact the elements in each partition should not change.

Cheers,
Till

On Wed, Mar 30, 2016 at 8:14 AM, Tarandeep Singh <[hidden email]> wrote:
It works in two phases/steps
1) Count number of elements in each partition (using mapPartition)
2) In second mapPartition, unique ID is assigned by calculating offset using number of elements computed in step 1.

Is there any chance the second mapPartition won't get same number of elements as first mapPartition (assuming data is in HDFS)?

Thanks
Tarandeep



Reply | Threaded
Open this post in threaded view
|

Re: DataSetUtils zipWithIndex question

Flavio Pompermaier
Ok, thanks for the clarification Till!

On Thu, Mar 31, 2016 at 2:14 PM, Till Rohrmann <[hidden email]> wrote:
A partition is the portion of data each task receives. Thus, the degree of parallelism of your program/task decides how many different partitions you have. Depending on the upstream operators (and which data is send to which task), the partitions will most likely differ in size.

Cheers,
Till

On Thu, Mar 31, 2016 at 2:11 PM, Flavio Pompermaier <[hidden email]> wrote:
Hi Till and Tarandeep,
I'm also interested in better understanding my knowledge about the concept of a partition..
From what I know a partition is the portion of data assigned by the job manager to each task manager..right?
Then, each partition is divided again at the task manager to maximize the slot usage..is it correct?
In every case, there will be a case where at least one partition is smaller than the others...am I wrong? Am I confusing some term..?

Best,
Flavio


On Thu, Mar 31, 2016 at 1:56 PM, Till Rohrmann <[hidden email]> wrote:
Hi Tarandeep,

the number of elements in each partition should stay constant. In fact the elements in each partition should not change.

Cheers,
Till

On Wed, Mar 30, 2016 at 8:14 AM, Tarandeep Singh <[hidden email]> wrote:
It works in two phases/steps
1) Count number of elements in each partition (using mapPartition)
2) In second mapPartition, unique ID is assigned by calculating offset using number of elements computed in step 1.

Is there any chance the second mapPartition won't get same number of elements as first mapPartition (assuming data is in HDFS)?

Thanks
Tarandeep