CsvInputFormat - read header line first

[madan]

Hi,

When we are splitting a csv file into multiple parts we are not sure which part is read first. Is there any way to make sure first part with header is read first ? I need to read header line first to store column name vs index and use this index for processing next records.

I could read header line from the file before submitting job to the flink, but that way we are opening the file 2 times. Is there any better way to do this? Please suggest.

--

[Ken Krugler]

Hi Madan,

If your source has a parallelism > 1, then when the CSV file is split, only one of the operators will get the split with the header row.

So in that case, how would you communicate the column name->index information to the other operators?

If you force a parallelism of 1 for the source, then I’m pretty sure you’re guaranteed that the file will be processed in order.

— Ken

On Oct 31, 2018, at 12:50 AM, madan <[hidden email]> wrote:

Hi,

When we are splitting a csv file into multiple parts we are not sure which part is read first. Is there any way to make sure first part with header is read first ? I need to read header line first to store column name vs index and use this index for processing next records.

I could read header line from the file before submitting job to the flink, but that way we are opening the file 2 times. Is there any better way to do this? Please suggest.

--

--------------------------

Ken Krugler
[phone]
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra

[madan]

Hi Ken,

Yep correct.

Thank you.

On Wed, Oct 31, 2018 at 7:24 PM Ken Krugler <[hidden email]> wrote:

Hi Madan,

If your source has a parallelism > 1, then when the CSV file is split, only one of the operators will get the split with the header row.

So in that case, how would you communicate the column name->index information to the other operators?

If you force a parallelism of 1 for the source, then I’m pretty sure you’re guaranteed that the file will be processed in order.

— Ken

On Oct 31, 2018, at 12:50 AM, madan <[hidden email]> wrote:

Hi,

When we are splitting a csv file into multiple parts we are not sure which part is read first. Is there any way to make sure first part with header is read first ? I need to read header line first to store column name vs index and use this index for processing next records.

I could read header line from the file before submitting job to the flink, but that way we are opening the file 2 times. Is there any better way to do this? Please suggest.

--

--------------------------

Ken Krugler
[phone]
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra

--