Most convenient data structure for unspecified length objects

[pietro]

I have to implement a program based on Flink that process some records.

The peculiarity of those records is that it is not possible to know at compile time how many fields they contain. Therefore, I cannot use a simple TupleN data type.

The solution I came up with, is to use a tuple with this structure:

(mandatory_field2, ..., mandatory_fieldM, Array[Int], Array[Double], Array[String] )

where the three arrays store the optional fields.

It worked, but is it a good way to do that?
Can I do it better?

Many thanks,
regards.

[Stephan Ewen]

Hi!

If you are programming in Scala, you can always use "Option[String]" for an optional String field.

Stephan

On Mon, Mar 16, 2015 at 4:57 PM, pietro <[hidden email]> wrote:

I have to implement a program based on Flink that process some records.

The peculiarity of those records is that it is not possible to know at
compile time how many fields they contain. Therefore, I cannot use a simple
TupleN data type.

The solution I came up with, is to use a tuple with this structure:

/(mandatory_field2, ..., mandatory_fieldM, Array[Int], Array[Double],
Array[String] )/

where the three arrays store the optional fields.

It worked, but is it a good way to do that?
Can I do it better?

Many thanks,
regards.




--
View this message in context: http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Most-convenient-data-structure-for-unspecified-length-objects-tp859.html
Sent from the Apache Flink (Incubator) User Mailing List archive. mailing list archive at Nabble.com.

[pietro]

Hi Stephan, thanks for the reply!

My problem is that I cannot know whether I will have 0, 1,2,..or more strings. Then, Option is not gonna help in my case :(

[Stephan Ewen]

Ah, okay. Then how about using a List of Strings?

On Mon, Mar 16, 2015 at 5:34 PM, pietro <[hidden email]> wrote:

Hi Stephan, thanks for the reply!

My problem is that I cannot know whether I will have 0, 1,2,..or more
strings. Then, Option is not gonna help in my case :(



--
View this message in context: http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Most-convenient-data-structure-for-unspecified-length-objects-tp859p861.html

Sent from the Apache Flink (Incubator) User Mailing List archive. mailing list archive at Nabble.com.

[Vinh June]

I ran into the same problem.
I think it depends on the input data, in my case it is CSV of unknown size. My solution is to read as text, then process on each line and add them into Map or Array of type Any