How to parse list values in csv file

[narasimha]

Hi,

Getting below error when trying to read a csv file, one of the field is list tupe 

Can someone help if fixing the issue 

jobmanager_1   | Caused by: java.lang.IllegalArgumentException: The type 'java.util.List' is not supported for the CSV input format.

jobmanager_1   | at org.apache.flink.api.common.io.GenericCsvInputFormat.setFieldsGeneric(GenericCsvInputFormat.java:289) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:67) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:83) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:87) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

--

[Yun Gao]

Hi,

    The CSV only supports the types listed in [1] and must use the types in this list, thus for other types some kind of workaround is needed, like first parsed as string and parsed again later in the program. 

Best,

Yun

[1] https://github.com/apache/flink/blob/e10e548feb2bedf54c3863bbd49ed4f9140546cf/flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java#L287

------------------Original Mail ------------------

Sender:narasimha <[hidden email]>

Send Date:Fri Dec 4 00:45:53 2020

Recipients:user <[hidden email]>

Subject:How to parse list values in csv file

Hi,

Getting below error when trying to read a csv file, one of the field is list tupe 

Can someone help if fixing the issue 

jobmanager_1   | Caused by: java.lang.IllegalArgumentException: The type 'java.util.List' is not supported for the CSV input format.

jobmanager_1   | at org.apache.flink.api.common.io.GenericCsvInputFormat.setFieldsGeneric(GenericCsvInputFormat.java:289) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:67) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:83) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:87) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

--

[narasimha]

thanks for you email.

Translated csv to JSON, read it as a plain text file and then processed to objects. 

It solved my use case. 

 

On Fri, Dec 4, 2020 at 12:24 PM Yun Gao <[hidden email]> wrote:

Hi,

    The CSV only supports the types listed in [1] and must use the types in this list, thus for other types some kind of workaround is needed, like first parsed as string and parsed again later in the program. 

Best,

Yun

[1] https://github.com/apache/flink/blob/e10e548feb2bedf54c3863bbd49ed4f9140546cf/flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java#L287

------------------Original Mail ------------------

Sender:narasimha <[hidden email]>

Send Date:Fri Dec 4 00:45:53 2020

Recipients:user <[hidden email]>

Subject:How to parse list values in csv file

Hi,

Getting below error when trying to read a csv file, one of the field is list tupe 

Can someone help if fixing the issue 

jobmanager_1   | Caused by: java.lang.IllegalArgumentException: The type 'java.util.List' is not supported for the CSV input format.

jobmanager_1   | at org.apache.flink.api.common.io.GenericCsvInputFormat.setFieldsGeneric(GenericCsvInputFormat.java:289) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:67) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:83) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:87) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

--

A.Narasimha Swamy

--

[Yun Gao]

Glad to hear that you solved this issue!

Best,

 Yun

------------------------------------------------------------------
Sender:narasimha<[hidden email]>
Date:2020/12/06 21:35:33
Recipient:Yun Gao<[hidden email]>
Cc:user<[hidden email]>
Theme:Re: How to parse list values in csv file

thanks for you email.

Translated csv to JSON, read it as a plain text file and then processed to objects. 

It solved my use case. 

 

On Fri, Dec 4, 2020 at 12:24 PM Yun Gao <[hidden email]> wrote:

Hi,

    The CSV only supports the types listed in [1] and must use the types in this list, thus for other types some kind of workaround is needed, like first parsed as string and parsed again later in the program. 

Best,

Yun

[1] https://github.com/apache/flink/blob/e10e548feb2bedf54c3863bbd49ed4f9140546cf/flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java#L287

------------------Original Mail ------------------

Sender:narasimha <[hidden email]>

Send Date:Fri Dec 4 00:45:53 2020

Recipients:user <[hidden email]>

Subject:How to parse list values in csv file

Hi,

Getting below error when trying to read a csv file, one of the field is list tupe 

Can someone help if fixing the issue 

jobmanager_1   | Caused by: java.lang.IllegalArgumentException: The type 'java.util.List' is not supported for the CSV input format.

jobmanager_1   | at org.apache.flink.api.common.io.GenericCsvInputFormat.setFieldsGeneric(GenericCsvInputFormat.java:289) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:67) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:83) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

jobmanager_1   | at org.apache.flink.api.java.io.RowCsvInputFormat.<init>(RowCsvInputFormat.java:87) ~[flink-dist_2.11-1.11.2.jar:1.11.2]

--

A.Narasimha Swamy

--