(DEPRECATED) Apache Flink User Mailing List archive.

Serializing NULLs

classic Classic list List threaded Threaded
♦
♦
 8 messages Options
Matt
Reply | Threaded
Open this post in threaded view
♦
♦
|

Serializing NULLs

Matt
Hello list,

I'm getting this error:

java.lang.RuntimeException: Could not forward element to next operator
...
Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of double of map in field properties of com.entities.Sector
...
Caused by: java.lang.NullPointerException

The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.

Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?

Regards,
Matt
Stephan Ewen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Serializing NULLs

Stephan Ewen
The "null" support in some types is not fully developed. However in that case I am wondering why it does not work. Can you share the stack trace, so we can take a look at the serializer?



On Mon, Dec 19, 2016 at 9:56 PM, Matt <[hidden email]> wrote:
Hello list,

I'm getting this error:

java.lang.RuntimeException: Could not forward element to next operator
...
Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of double of map in field properties of com.entities.Sector
...
Caused by: java.lang.NullPointerException

The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.

Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?

Regards,
Matt
Matt
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Serializing NULLs

Matt

In the meanwhile I've solved the issue by creating a POJO class where null is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess it's not as fast as de/serializing Double though.

If you need any other information let me know.

Regards,
Matt

On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <[hidden email]> wrote:
The "null" support in some types is not fully developed. However in that case I am wondering why it does not work. Can you share the stack trace, so we can take a look at the serializer?



On Mon, Dec 19, 2016 at 9:56 PM, Matt <[hidden email]> wrote:
Hello list,

I'm getting this error:

java.lang.RuntimeException: Could not forward element to next operator
...
Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of double of map in field properties of com.entities.Sector
...
Caused by: java.lang.NullPointerException

The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.

Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?

Regards,
Matt
Deepak Sharma
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Serializing NULLs

Deepak Sharma
Hi Matt
Can you please share the code like how you are mapping null to long?

Thanks
Deepak

On Tue, Dec 20, 2016 at 9:25 PM, Matt <[hidden email]> wrote:

In the meanwhile I've solved the issue by creating a POJO class where null is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess it's not as fast as de/serializing Double though.

If you need any other information let me know.

Regards,
Matt

On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <[hidden email]> wrote:
The "null" support in some types is not fully developed. However in that case I am wondering why it does not work. Can you share the stack trace, so we can take a look at the serializer?



On Mon, Dec 19, 2016 at 9:56 PM, Matt <[hidden email]> wrote:
Hello list,

I'm getting this error:

java.lang.RuntimeException: Could not forward element to next operator
...
Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of double of map in field properties of com.entities.Sector
...
Caused by: java.lang.NullPointerException

The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.

Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?

Regards,
Matt





--
Stephan Ewen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Serializing NULLs

Stephan Ewen
In reply to this post by Matt
Thanks for sharing the stack trace.

This seems not really Flink related, it is part of the specific Avro encoding logic.
The Avro Generic Record Type apparently does not allow the map value to be null.



On Tue, Dec 20, 2016 at 4:55 PM, Matt <[hidden email]> wrote:

In the meanwhile I've solved the issue by creating a POJO class where null is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess it's not as fast as de/serializing Double though.

If you need any other information let me know.

Regards,
Matt

On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <[hidden email]> wrote:
The "null" support in some types is not fully developed. However in that case I am wondering why it does not work. Can you share the stack trace, so we can take a look at the serializer?



On Mon, Dec 19, 2016 at 9:56 PM, Matt <[hidden email]> wrote:
Hello list,

I'm getting this error:

java.lang.RuntimeException: Could not forward element to next operator
...
Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of double of map in field properties of com.entities.Sector
...
Caused by: java.lang.NullPointerException

The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.

Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?

Regards,
Matt
Anirudh Mallem
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Serializing NULLs

Anirudh Mallem
If you are using Avro generated classes then you cannot have your values null. 

From: Stephan Ewen
Reply-To: "[hidden email]"
Date: Tuesday, December 20, 2016 at 8:17 AM
To: "[hidden email]"
Subject: Re: Serializing NULLs

Thanks for sharing the stack trace.

This seems not really Flink related, it is part of the specific Avro encoding logic.
The Avro Generic Record Type apparently does not allow the map value to be null.



On Tue, Dec 20, 2016 at 4:55 PM, Matt <[hidden email]> wrote:

In the meanwhile I've solved the issue by creating a POJO class where null is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess it's not as fast as de/serializing Double though.

If you need any other information let me know.

Regards,
Matt

On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <[hidden email]> wrote:
The "null" support in some types is not fully developed. However in that case I am wondering why it does not work. Can you share the stack trace, so we can take a look at the serializer?



On Mon, Dec 19, 2016 at 9:56 PM, Matt <[hidden email]> wrote:
Hello list,

I'm getting this error:

java.lang.RuntimeException: Could not forward element to next operator
...
Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of double of map in field properties of com.entities.Sector
...
Caused by: java.lang.NullPointerException

The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.

Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?

Regards,
Matt
Matt
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Serializing NULLs

Matt
Here is the code of a Double wrapper with null support [1].


On Tue, Dec 20, 2016 at 4:26 PM, Anirudh Mallem <[hidden email]> wrote:
If you are using Avro generated classes then you cannot have your values null. 
<a href="https://cwiki.apache.org/confluence/display/AVRO/FAQ#FAQ-Whyisn&#39;teveryvalueinAvronullable?" class="m_-8368338872522317348toc-link" style="color:rgb(50,108,166);text-decoration:none" target="_blank">https://cwiki.apache.org/confluence/display/AVRO/FAQ#FAQ-Whyisn'teveryvalueinAvronullable?

From: Stephan Ewen
Reply-To: "[hidden email]"
Date: Tuesday, December 20, 2016 at 8:17 AM
To: "[hidden email]"
Subject: Re: Serializing NULLs

Thanks for sharing the stack trace.

This seems not really Flink related, it is part of the specific Avro encoding logic.
The Avro Generic Record Type apparently does not allow the map value to be null.



On Tue, Dec 20, 2016 at 4:55 PM, Matt <[hidden email]> wrote:

In the meanwhile I've solved the issue by creating a POJO class where null is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess it's not as fast as de/serializing Double though.

If you need any other information let me know.

Regards,
Matt

On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <[hidden email]> wrote:
The "null" support in some types is not fully developed. However in that case I am wondering why it does not work. Can you share the stack trace, so we can take a look at the serializer?



On Mon, Dec 19, 2016 at 9:56 PM, Matt <[hidden email]> wrote:
Hello list,

I'm getting this error:

java.lang.RuntimeException: Could not forward element to next operator
...
Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of double of map in field properties of com.entities.Sector
...
Caused by: java.lang.NullPointerException

The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.

Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?

Regards,
Matt

Newport, Billy
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: Serializing NULLs

Newport, Billy
In reply to this post by Anirudh Mallem

Map<union<null,double>> in your avro schema is what you want here if the map values are nullable.

 

 

From: Anirudh Mallem [mailto:[hidden email]]
Sent: Tuesday, December 20, 2016 2:26 PM
To: [hidden email]
Subject: Re: Serializing NULLs

 

If you are using Avro generated classes then you cannot have your values null. 

 

From: Stephan Ewen
Reply-To: "[hidden email]"
Date: Tuesday, December 20, 2016 at 8:17 AM
To: "[hidden email]"
Subject: Re: Serializing NULLs

 

Thanks for sharing the stack trace.

 

This seems not really Flink related, it is part of the specific Avro encoding logic.

The Avro Generic Record Type apparently does not allow the map value to be null.

 

 

 

On Tue, Dec 20, 2016 at 4:55 PM, Matt <[hidden email]> wrote:

 

In the meanwhile I've solved the issue by creating a POJO class where null is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess it's not as fast as de/serializing Double though.

 

If you need any other information let me know.

 

Regards,

Matt

 

On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <[hidden email]> wrote:

The "null" support in some types is not fully developed. However in that case I am wondering why it does not work. Can you share the stack trace, so we can take a look at the serializer?

 

 

 

On Mon, Dec 19, 2016 at 9:56 PM, Matt <[hidden email]> wrote:

Hello list,

 

I'm getting this error:

 

java.lang.RuntimeException: Could not forward element to next operator

...

Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of double of map in field properties of com.entities.Sector

...

Caused by: java.lang.NullPointerException

 

The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.

 

Why isn't it possible to forward/serialize those elements with null values?

What do you do when your elements may contain nulls?

 

Regards,

Matt

 

 

 
Free forum by Nabble Edit this page