Bug in Table api CsvTableSink

[Flavio Pompermaier]

Hi to all,

I'm trying to read from a db and then writing to a csv.

In my code I do the following:

tableEnv.fromDataSet(myDataSet).writeToSink(new CsvTableSink(csvOutputDir, fieldDelim));

If I use fieldDelim= "," everything is Ok, if I use "\t" some tab is not printed correctly...

PS: myDataSet is a dataset of 32 String fields.

Is is something that has been fixed in Flink > 1.1.1?

Best,

Flavio

[Fabian Hueske-2]

Hi Flavio,

I do not remember that such a bug was fixed. Maybe by chance, but I guess not.

Can you open a JIRA and maybe provide input data to reproduce the problem?

Thank you,

Fabian

2017-01-31 16:25 GMT+01:00 Flavio Pompermaier <[hidden email]>:

Hi to all,

I'm trying to read from a db and then writing to a csv.

In my code I do the following:

tableEnv.fromDataSet(myDataSet).writeToSink(new CsvTableSink(csvOutputDir, fieldDelim));

If I use fieldDelim= "," everything is Ok, if I use "\t" some tab is not printed correctly...

PS: myDataSet is a dataset of 32 String fields.

Is is something that has been fixed in Flink > 1.1.1?

Best,

Flavio

[Flavio Pompermaier]

I hope to have time to write a test program :)

Otherwise I hope someone else could give it a try in the meantime..

Best,

Flavio

On Tue, Jan 31, 2017 at 4:49 PM, Fabian Hueske <[hidden email]> wrote:

Hi Flavio,

I do not remember that such a bug was fixed. Maybe by chance, but I guess not.

Can you open a JIRA and maybe provide input data to reproduce the problem?

Thank you,

Fabian

2017-01-31 16:25 GMT+01:00 Flavio Pompermaier <[hidden email]>:

Hi to all,

I'm trying to read from a db and then writing to a csv.

In my code I do the following:

tableEnv.fromDataSet(myDataSet).writeToSink(new CsvTableSink(csvOutputDir, fieldDelim));

If I use fieldDelim= "," everything is Ok, if I use "\t" some tab is not printed correctly...

PS: myDataSet is a dataset of 32 String fields.

Is is something that has been fixed in Flink > 1.1.1?

Best,

Flavio

[Flavio Pompermaier]

These 2 rows if converted to Row[] of Strings should cause the problem:

http://www.aaa.it/xxx/v/10002780063,,,,t/0000000/1,f/10001957530,cf/13,cpva/77,cf/13,,sit/A2046X,strp/408,10921957530,,1,5,1,2013-01-04T15:02:25,5,,10002780063,XXXXXXX,1,,3,,,2013-01-04T15:02:25,XXXXXXX,XXXXXXX,13,2013-01-04T15:02:25

http://www.aaa.it/xxx/v/10002780063,,,,t/0000000/1,f/10004002060,cf/3,cpva/7,cf/3,f/10164002060,sit/A15730L,strp/408,10164002060,10164002060,2,7,1,2008-05-29T11:47:35,1,,10002780063,XXXXXXX,1,,0,,,2008-05-29T11:47:35,XXXXXXX,XXXXXXX,3,2008-05-29T11:47:35

Best,

Flavio

On Tue, Jan 31, 2017 at 4:51 PM, Flavio Pompermaier <[hidden email]> wrote:

I hope to have time to write a test program :)

Otherwise I hope someone else could give it a try in the meantime..

Best,

Flavio

On Tue, Jan 31, 2017 at 4:49 PM, Fabian Hueske <[hidden email]> wrote:

Hi Flavio,

I do not remember that such a bug was fixed. Maybe by chance, but I guess not.

Can you open a JIRA and maybe provide input data to reproduce the problem?

Thank you,

Fabian

2017-01-31 16:25 GMT+01:00 Flavio Pompermaier <[hidden email]>:

Hi to all,

I'm trying to read from a db and then writing to a csv.

In my code I do the following:

tableEnv.fromDataSet(myDataSet).writeToSink(new CsvTableSink(csvOutputDir, fieldDelim));

If I use fieldDelim= "," everything is Ok, if I use "\t" some tab is not printed correctly...

PS: myDataSet is a dataset of 32 String fields.

Is is something that has been fixed in Flink > 1.1.1?

Best,

Flavio

[Fabian Hueske-2]

Hi Flavio,

I checked the records on the current master and the CsvTableSink seemed to work fine.

However, I had some issues when converting the DataSet[Row] into a table.

You have to make sure that the TypeInformation for DataSet[Row] is RowTypeInfo and not GenericType[Row].

Can you check the type of your myDataSet?

Best, Fabian

2017-01-31 17:51 GMT+01:00 Flavio Pompermaier <[hidden email]>:

These 2 rows if converted to Row[] of Strings should cause the problem:

http://www.aaa.it/xxx/v/10002780063,,,,t/0000000/1,f/10001957530,cf/13,cpva/77,cf/13,,sit/A2046X,strp/408,10921957530,,1,5,1,2013-01-04T15:02:25,5,,10002780063,XXXXXXX,1,,3,,,2013-01-04T15:02:25,XXXXXXX,XXXXXXX,13,2013-01-04T15:02:25

http://www.aaa.it/xxx/v/10002780063,,,,t/0000000/1,f/10004002060,cf/3,cpva/7,cf/3,f/10164002060,sit/A15730L,strp/408,10164002060,10164002060,2,7,1,2008-05-29T11:47:35,1,,10002780063,XXXXXXX,1,,0,,,2008-05-29T11:47:35,XXXXXXX,XXXXXXX,3,2008-05-29T11:47:35

Best,

Flavio

On Tue, Jan 31, 2017 at 4:51 PM, Flavio Pompermaier <[hidden email]> wrote:

I hope to have time to write a test program :)

Otherwise I hope someone else could give it a try in the meantime..

Best,

Flavio

On Tue, Jan 31, 2017 at 4:49 PM, Fabian Hueske <[hidden email]> wrote:

Hi Flavio,

I do not remember that such a bug was fixed. Maybe by chance, but I guess not.

Can you open a JIRA and maybe provide input data to reproduce the problem?

Thank you,

Fabian

2017-01-31 16:25 GMT+01:00 Flavio Pompermaier <[hidden email]>:

Hi to all,

I'm trying to read from a db and then writing to a csv.

In my code I do the following:

tableEnv.fromDataSet(myDataSet).writeToSink(new CsvTableSink(csvOutputDir, fieldDelim));

If I use fieldDelim= "," everything is Ok, if I use "\t" some tab is not printed correctly...

PS: myDataSet is a dataset of 32 String fields.

Is is something that has been fixed in Flink > 1.1.1?

Best,

Flavio

[Flavio Pompermaier]

The error was just caused by a cut-and-paste on the shell instead of opening directly the file so bug solved :)

On Mon, Feb 6, 2017 at 12:20 PM, Fabian Hueske <[hidden email]> wrote:

Hi Flavio,

I checked the records on the current master and the CsvTableSink seemed to work fine.

However, I had some issues when converting the DataSet[Row] into a table.

You have to make sure that the TypeInformation for DataSet[Row] is RowTypeInfo and not GenericType[Row].

Can you check the type of your myDataSet?

Best, Fabian

2017-01-31 17:51 GMT+01:00 Flavio Pompermaier <[hidden email]>:

These 2 rows if converted to Row[] of Strings should cause the problem:

http://www.aaa.it/xxx/v/10002780063,,,,t/0000000/1,f/10001957530,cf/13,cpva/77,cf/13,,sit/A2046X,strp/408,10921957530,,1,5,1,2013-01-04T15:02:25,5,,10002780063,XXXXXXX,1,,3,,,2013-01-04T15:02:25,XXXXXXX,XXXXXXX,13,2013-01-04T15:02:25

http://www.aaa.it/xxx/v/10002780063,,,,t/0000000/1,f/10004002060,cf/3,cpva/7,cf/3,f/10164002060,sit/A15730L,strp/408,10164002060,10164002060,2,7,1,2008-05-29T11:47:35,1,,10002780063,XXXXXXX,1,,0,,,2008-05-29T11:47:35,XXXXXXX,XXXXXXX,3,2008-05-29T11:47:35

Best,

Flavio

On Tue, Jan 31, 2017 at 4:51 PM, Flavio Pompermaier <[hidden email]> wrote:

I hope to have time to write a test program :)

Otherwise I hope someone else could give it a try in the meantime..

Best,

Flavio

On Tue, Jan 31, 2017 at 4:49 PM, Fabian Hueske <[hidden email]> wrote:

Hi Flavio,

I do not remember that such a bug was fixed. Maybe by chance, but I guess not.

Can you open a JIRA and maybe provide input data to reproduce the problem?

Thank you,

Fabian

2017-01-31 16:25 GMT+01:00 Flavio Pompermaier <[hidden email]>:

Hi to all,

I'm trying to read from a db and then writing to a csv.

In my code I do the following:

tableEnv.fromDataSet(myDataSet).writeToSink(new CsvTableSink(csvOutputDir, fieldDelim));

If I use fieldDelim= "," everything is Ok, if I use "\t" some tab is not printed correctly...

PS: myDataSet is a dataset of 32 String fields.

Is is something that has been fixed in Flink > 1.1.1?

Best,

Flavio