keyBy on a collection of Pojos

[Rami Al-Isawi]

Hi,

I was trying to test some specific issue, but now I cannot seem to get the very basic case working. It is most likely that I am blind to something, would anyone have quick look at it?

https://gist.github.com/rami-alisawi/d6ff33ae2d4d6e7bb1f8b329e3e5fa77

It is just a collection of pojos where I am just trying to keyBy one field and sum into the other, but I am getting:

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA’}

Regards,

-Rami

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

[Flavio Pompermaier]

Conditions for a class to be treated as a POJO by Flink:

• The class must be public
• It must have a public constructor without arguments
• All fields either have to be public or there must be getters and setters for all non-public fields. If the field name is foo the getter and setters must be called getFoo() and setFoo().

I don't know whether you need to implement also hashCode() and equals() actually

Best,

Flavio

On Mon, May 23, 2016 at 3:24 PM, Al-Isawi Rami <[hidden email]> wrote:

Hi,

I was trying to test some specific issue, but now I cannot seem to get the very basic case working. It is most likely that I am blind to something, would anyone have quick look at it?

https://gist.github.com/rami-alisawi/d6ff33ae2d4d6e7bb1f8b329e3e5fa77

It is just a collection of pojos where I am just trying to keyBy one field and sum into the other, but I am getting:

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA’}

Regards,

-Rami

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

[Rami Al-Isawi]

Thanks Flavio, but as you can see in my code I have already declared my pojo to achieve those conditions:

public class PojoExample {
	public int count;
	public String productId;
	public PojoExample() {
	}

}

So it cannot be that.

-Rami

On 23 May 2016, at 16:30, Flavio Pompermaier <[hidden email]> wrote:

Conditions for a class to be treated as a POJO by Flink:

• The class must be public
• It must have a public constructor without arguments
• All fields either have to be public or there must be getters and setters for all non-public fields. If the field name is foo the getter and setters must be called getFoo() and setFoo().

I don't know whether you need to implement also hashCode() and equals() actually

Best,

Flavio

On Mon, May 23, 2016 at 3:24 PM, Al-Isawi Rami <[hidden email]> wrote:

Hi,

I was trying to test some specific issue, but now I cannot seem to get the very basic case working. It is most likely that I am blind to something, would anyone have quick look at it?

https://gist.github.com/rami-alisawi/d6ff33ae2d4d6e7bb1f8b329e3e5fa77

It is just a collection of pojos where I am just trying to keyBy one field and sum into the other, but I am getting:

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA’}

Regards,

-Rami

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

[Flavio Pompermaier]

You don't have getters and setters for count and productId.

Your class should be

public class PojoExample {

public int count;

public String productId;

public PojoExample() {}

public int getCount() {

return count;

}

public void setCount(int count) {

this.count = count;

}

public String getProductId() {

return productId;

}

public void setProductId(String productId) {

this.productId = productId;

}

}

On Mon, May 23, 2016 at 3:40 PM, Al-Isawi Rami <[hidden email]> wrote:

Thanks Flavio, but as you can see in my code I have already declared my pojo to achieve those conditions:

public class PojoExample {
	public int count;
	public String productId;
	public PojoExample() {
	}

}

So it cannot be that.

-Rami

On 23 May 2016, at 16:30, Flavio Pompermaier <[hidden email]> wrote:

Conditions for a class to be treated as a POJO by Flink:

• The class must be public
• It must have a public constructor without arguments
• All fields either have to be public or there must be getters and setters for all non-public fields. If the field name is foo the getter and setters must be called getFoo() and setFoo().

I don't know whether you need to implement also hashCode() and equals() actually

Best,

Flavio

On Mon, May 23, 2016 at 3:24 PM, Al-Isawi Rami <[hidden email]> wrote:

Hi,

I was trying to test some specific issue, but now I cannot seem to get the very basic case working. It is most likely that I am blind to something, would anyone have quick look at it?

https://gist.github.com/rami-alisawi/d6ff33ae2d4d6e7bb1f8b329e3e5fa77

It is just a collection of pojos where I am just trying to keyBy one field and sum into the other, but I am getting:

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA’}

Regards,

-Rami

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

[Rami Al-Isawi]

Thanks, setters and getters for public fields have no purpose. Also per the conditions you have mentioned:

"All fields either have to be public or there must be getters and setters for all non-public fields.”

Since my fields are declared public there are no impact on adding getters and setters. ( I have also testing after adding the setters and getters and as expected that has no effect).

Could you spot anything else? this should be really easy basic case. I am really wondering why it is not working.

For the people who are lazy to open the gist code snippet, this is what I am trying to do:

pojoExampleDataStream.
	keyBy("productId").
	sum("count").
	print();

Regards,

-Rami

On 23 May 2016, at 17:11, Flavio Pompermaier <[hidden email]> wrote:

You don't have getters and setters for count and productId.

Your class should be

public class PojoExample {

public int count;

public String productId;

public PojoExample() {}

public int getCount() {

return count;

}

public void setCount(int count) {

this.count = count;

}

public String getProductId() {

return productId;

}

public void setProductId(String productId) {

this.productId = productId;

}

}

On Mon, May 23, 2016 at 3:40 PM, Al-Isawi Rami <[hidden email]> wrote:

Thanks Flavio, but as you can see in my code I have already declared my pojo to achieve those conditions:

public class PojoExample {
	public int count;
	public String productId;
	public PojoExample() {
	}

}

So it cannot be that.

-Rami

On 23 May 2016, at 16:30, Flavio Pompermaier <[hidden email]> wrote:

Conditions for a class to be treated as a POJO by Flink:

• The class must be public
• It must have a public constructor without arguments
• All fields either have to be public or there must be getters and setters for all non-public fields. If the field name is foo the getter and setters must be called getFoo() and setFoo().

I don't know whether you need to implement also hashCode() and equals() actually

Best,

Flavio

On Mon, May 23, 2016 at 3:24 PM, Al-Isawi Rami <[hidden email]> wrote:

Hi,

I was trying to test some specific issue, but now I cannot seem to get the very basic case working. It is most likely that I am blind to something, would anyone have quick look at it?

https://gist.github.com/rami-alisawi/d6ff33ae2d4d6e7bb1f8b329e3e5fa77

It is just a collection of pojos where I am just trying to keyBy one field and sum into the other, but I am getting:

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA’}

Regards,

-Rami

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

[Flavio Pompermaier]

Sorry Rami, you're right :)

Unfortunattely I've never used Flink streaming so I cannot be helpful there..

Myabe is it something related to the default triggering policy of the streaming environment?

On Mon, May 23, 2016 at 5:06 PM, Al-Isawi Rami <[hidden email]> wrote:

Thanks, setters and getters for public fields have no purpose. Also per the conditions you have mentioned:

"All fields either have to be public or there must be getters and setters for all non-public fields.”

Since my fields are declared public there are no impact on adding getters and setters. ( I have also testing after adding the setters and getters and as expected that has no effect).

Could you spot anything else? this should be really easy basic case. I am really wondering why it is not working.

For the people who are lazy to open the gist code snippet, this is what I am trying to do:

pojoExampleDataStream.
	keyBy("productId").
	sum("count").
	print();

Regards,

-Rami

On 23 May 2016, at 17:11, Flavio Pompermaier <[hidden email]> wrote:

You don't have getters and setters for count and productId.

Your class should be

public class PojoExample {

public int count;

public String productId;

public PojoExample() {}

public int getCount() {

return count;

}

public void setCount(int count) {

this.count = count;

}

public String getProductId() {

return productId;

}

public void setProductId(String productId) {

this.productId = productId;

}

}

On Mon, May 23, 2016 at 3:40 PM, Al-Isawi Rami <[hidden email]> wrote:

Thanks Flavio, but as you can see in my code I have already declared my pojo to achieve those conditions:

public class PojoExample {
	public int count;
	public String productId;
	public PojoExample() {
	}

}

So it cannot be that.

-Rami

On 23 May 2016, at 16:30, Flavio Pompermaier <[hidden email]> wrote:

Conditions for a class to be treated as a POJO by Flink:

• The class must be public
• It must have a public constructor without arguments
• All fields either have to be public or there must be getters and setters for all non-public fields. If the field name is foo the getter and setters must be called getFoo() and setFoo().

I don't know whether you need to implement also hashCode() and equals() actually

Best,

Flavio

On Mon, May 23, 2016 at 3:24 PM, Al-Isawi Rami <[hidden email]> wrote:

Hi,

I was trying to test some specific issue, but now I cannot seem to get the very basic case working. It is most likely that I am blind to something, would anyone have quick look at it?

https://gist.github.com/rami-alisawi/d6ff33ae2d4d6e7bb1f8b329e3e5fa77

It is just a collection of pojos where I am just trying to keyBy one field and sum into the other, but I am getting:

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA’}

Regards,

-Rami

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

[Deepak Sharma]

Can you try serializing your POJO ?

Thanks

Deepak

On Mon, May 23, 2016 at 8:42 PM, Flavio Pompermaier <[hidden email]> wrote:

Sorry Rami, you're right :)

Unfortunattely I've never used Flink streaming so I cannot be helpful there..

Myabe is it something related to the default triggering policy of the streaming environment?

On Mon, May 23, 2016 at 5:06 PM, Al-Isawi Rami <[hidden email]> wrote:

Thanks, setters and getters for public fields have no purpose. Also per the conditions you have mentioned:

"All fields either have to be public or there must be getters and setters for all non-public fields.”

Since my fields are declared public there are no impact on adding getters and setters. ( I have also testing after adding the setters and getters and as expected that has no effect).

Could you spot anything else? this should be really easy basic case. I am really wondering why it is not working.

For the people who are lazy to open the gist code snippet, this is what I am trying to do:

pojoExampleDataStream.
	keyBy("productId").
	sum("count").
	print();

Regards,

-Rami

On 23 May 2016, at 17:11, Flavio Pompermaier <[hidden email]> wrote:

You don't have getters and setters for count and productId.

Your class should be

public class PojoExample {

public int count;

public String productId;

public PojoExample() {}

public int getCount() {

return count;

}

public void setCount(int count) {

this.count = count;

}

public String getProductId() {

return productId;

}

public void setProductId(String productId) {

this.productId = productId;

}

}

On Mon, May 23, 2016 at 3:40 PM, Al-Isawi Rami <[hidden email]> wrote:

Thanks Flavio, but as you can see in my code I have already declared my pojo to achieve those conditions:

public class PojoExample {
	public int count;
	public String productId;
	public PojoExample() {
	}

}

So it cannot be that.

-Rami

On 23 May 2016, at 16:30, Flavio Pompermaier <[hidden email]> wrote:

Conditions for a class to be treated as a POJO by Flink:

• The class must be public
• It must have a public constructor without arguments
• All fields either have to be public or there must be getters and setters for all non-public fields. If the field name is foo the getter and setters must be called getFoo() and setFoo().

I don't know whether you need to implement also hashCode() and equals() actually

Best,

Flavio

On Mon, May 23, 2016 at 3:24 PM, Al-Isawi Rami <[hidden email]> wrote:

Hi,

I was trying to test some specific issue, but now I cannot seem to get the very basic case working. It is most likely that I am blind to something, would anyone have quick look at it?

https://gist.github.com/rami-alisawi/d6ff33ae2d4d6e7bb1f8b329e3e5fa77

It is just a collection of pojos where I am just trying to keyBy one field and sum into the other, but I am getting:

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA’}

Regards,

-Rami

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

--

[Fabian Hueske-2]

Actually, the program works correctly (according to the DataStream API)

Let me explain what happens:

1) You do not initialize the count variable, so it will be 0 (summing 0s results in 0)

2) DataStreams are considered to be unbound (have an infinite size). KeyBy does not group the records because it would have to wait forever to close the group due to the infinite input. Instead keyBy basically partitions the data.

3) By calling sum() on a KeyedStream you compute a running aggregate which emits one record for each incoming record summing the declared field (this stays 0 because 0 + 0 = 0).

You will need to

1) initialize count to 1

2) define window to discretize the stream into finite sets (windows) of records.

Cheers, Fabian

2016-05-23 17:16 GMT+02:00 Deepak Sharma <[hidden email]>:

Can you try serializing your POJO ?

Thanks

Deepak

On Mon, May 23, 2016 at 8:42 PM, Flavio Pompermaier <[hidden email]> wrote:

Sorry Rami, you're right :)

Unfortunattely I've never used Flink streaming so I cannot be helpful there..

Myabe is it something related to the default triggering policy of the streaming environment?

On Mon, May 23, 2016 at 5:06 PM, Al-Isawi Rami <[hidden email]> wrote:

Thanks, setters and getters for public fields have no purpose. Also per the conditions you have mentioned:

"All fields either have to be public or there must be getters and setters for all non-public fields.”

Since my fields are declared public there are no impact on adding getters and setters. ( I have also testing after adding the setters and getters and as expected that has no effect).

Could you spot anything else? this should be really easy basic case. I am really wondering why it is not working.

For the people who are lazy to open the gist code snippet, this is what I am trying to do:

pojoExampleDataStream.
	keyBy("productId").
	sum("count").
	print();

Regards,

-Rami

On 23 May 2016, at 17:11, Flavio Pompermaier <[hidden email]> wrote:

You don't have getters and setters for count and productId.

Your class should be

public class PojoExample {

public int count;

public String productId;

public PojoExample() {}

public int getCount() {

return count;

}

public void setCount(int count) {

this.count = count;

}

public String getProductId() {

return productId;

}

public void setProductId(String productId) {

this.productId = productId;

}

}

On Mon, May 23, 2016 at 3:40 PM, Al-Isawi Rami <[hidden email]> wrote:

Thanks Flavio, but as you can see in my code I have already declared my pojo to achieve those conditions:

public class PojoExample {
	public int count;
	public String productId;
	public PojoExample() {
	}

}

So it cannot be that.

-Rami

On 23 May 2016, at 16:30, Flavio Pompermaier <[hidden email]> wrote:

Conditions for a class to be treated as a POJO by Flink:

• The class must be public
• It must have a public constructor without arguments
• All fields either have to be public or there must be getters and setters for all non-public fields. If the field name is foo the getter and setters must be called getFoo() and setFoo().

I don't know whether you need to implement also hashCode() and equals() actually

Best,

Flavio

On Mon, May 23, 2016 at 3:24 PM, Al-Isawi Rami <[hidden email]> wrote:

Hi,

I was trying to test some specific issue, but now I cannot seem to get the very basic case working. It is most likely that I am blind to something, would anyone have quick look at it?

https://gist.github.com/rami-alisawi/d6ff33ae2d4d6e7bb1f8b329e3e5fa77

It is just a collection of pojos where I am just trying to keyBy one field and sum into the other, but I am getting:

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

8> PojoExample{count=0, productId='productB'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA'}

5> PojoExample{count=0, productId='productA’}

Regards,

-Rami

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.

--

Thanks
Deepak
www.bigdatabig.com
www.keosha.net