Read CSV Parse Quoted Strings Function

[Tamara Mendt]

Hi all,

When using the parseQuotedStrings function for the CsvReader class, I have noticed that if the caracter of the quotes is also inside of the string, the parsing fails.

For example, if there is a field of this form:

"RT @sportsguy33: New Time Warner slogan: "Time Warner, where we make you long for the days before cable.""

I think it is not so uncommon to have a case like this and it should not fail, but rather the string should be parsed as:

RT @sportsguy33: New Time Warner slogan: "Time Warner, where we make you long for the days before cable."

I have found the part of the Flink code that raised this exception and can fix it, but wanted to consult first if others agree that this is an issue.

Cheers,

Tamara

[Maximilian Michels]

Hi Tamara,

Quoted strings should not contain the quoting character. The way to work around this is to escape the quote characters. However, currently there is no option to escape quotes which pretty much forbids any use of quote characters within quoted fields. This should be fixed. I opened a JIRA for this issue: https://issues.apache.org/jira/browse/FLINK-2567

As for your idea for parsing quoted fields, I personally prefer escaping the quoting characters. In quoted fields, Flink allows all characters except quotes which means, we have to read the entire file to know whether we can close a quote. Additionally, we need to keep track of how many quotes are opened and closed.

While your proposal is a very convenient feature, I think we should rather implement explicit quoting for performance and clarity reasons.

Cheers,

Max

On Mon, Aug 24, 2015 at 10:40 AM, Tamara Mendt <[hidden email]> wrote:

Hi all,

When using the parseQuotedStrings function for the CsvReader class, I have noticed that if the caracter of the quotes is also inside of the string, the parsing fails.

For example, if there is a field of this form:

"RT @sportsguy33: New Time Warner slogan: "Time Warner, where we make you long for the days before cable.""

I think it is not so uncommon to have a case like this and it should not fail, but rather the string should be parsed as:

RT @sportsguy33: New Time Warner slogan: "Time Warner, where we make you long for the days before cable."

I have found the part of the Flink code that raised this exception and can fix it, but wanted to consult first if others agree that this is an issue.

Cheers,

Tamara

[Tamara Mendt]

Thank you Maximilian,

I agree and would be happy to fix this issue.

Cheers,

Tamara.

On Mon, Aug 24, 2015 at 11:50 AM, Maximilian Michels <[hidden email]> wrote:

Hi Tamara,

Quoted strings should not contain the quoting character. The way to work around this is to escape the quote characters. However, currently there is no option to escape quotes which pretty much forbids any use of quote characters within quoted fields. This should be fixed. I opened a JIRA for this issue: https://issues.apache.org/jira/browse/FLINK-2567

As for your idea for parsing quoted fields, I personally prefer escaping the quoting characters. In quoted fields, Flink allows all characters except quotes which means, we have to read the entire file to know whether we can close a quote. Additionally, we need to keep track of how many quotes are opened and closed.

While your proposal is a very convenient feature, I think we should rather implement explicit quoting for performance and clarity reasons.

Cheers,

Max

On Mon, Aug 24, 2015 at 10:40 AM, Tamara Mendt <[hidden email]> wrote:

Hi all,

When using the parseQuotedStrings function for the CsvReader class, I have noticed that if the caracter of the quotes is also inside of the string, the parsing fails.

For example, if there is a field of this form:

"RT @sportsguy33: New Time Warner slogan: "Time Warner, where we make you long for the days before cable.""

I think it is not so uncommon to have a case like this and it should not fail, but rather the string should be parsed as:

RT @sportsguy33: New Time Warner slogan: "Time Warner, where we make you long for the days before cable."

I have found the part of the Flink code that raised this exception and can fix it, but wanted to consult first if others agree that this is an issue.

Cheers,

Tamara

--

[Stephan Ewen]

Be aware that the CSV input format extends the delimited input format. The delimited input format splits at the line delimiter (such as \n) without awareness of quotes. So that character can never be part of a quote...

On Mon, Aug 24, 2015 at 11:55 AM, Tamara Mendt <[hidden email]> wrote:

Thank you Maximilian,

I agree and would be happy to fix this issue.

Cheers,

Tamara.

On Mon, Aug 24, 2015 at 11:50 AM, Maximilian Michels <[hidden email]> wrote:

Hi Tamara,

Quoted strings should not contain the quoting character. The way to work around this is to escape the quote characters. However, currently there is no option to escape quotes which pretty much forbids any use of quote characters within quoted fields. This should be fixed. I opened a JIRA for this issue: https://issues.apache.org/jira/browse/FLINK-2567

As for your idea for parsing quoted fields, I personally prefer escaping the quoting characters. In quoted fields, Flink allows all characters except quotes which means, we have to read the entire file to know whether we can close a quote. Additionally, we need to keep track of how many quotes are opened and closed.

While your proposal is a very convenient feature, I think we should rather implement explicit quoting for performance and clarity reasons.

Cheers,

Max

On Mon, Aug 24, 2015 at 10:40 AM, Tamara Mendt <[hidden email]> wrote:

Hi all,

When using the parseQuotedStrings function for the CsvReader class, I have noticed that if the caracter of the quotes is also inside of the string, the parsing fails.

For example, if there is a field of this form:

"RT @sportsguy33: New Time Warner slogan: "Time Warner, where we make you long for the days before cable.""

I think it is not so uncommon to have a case like this and it should not fail, but rather the string should be parsed as:

RT @sportsguy33: New Time Warner slogan: "Time Warner, where we make you long for the days before cable."

I have found the part of the Flink code that raised this exception and can fix it, but wanted to consult first if others agree that this is an issue.

Cheers,

Tamara

--

Tamara Mendt