IOException reading a large file

[Flavio Pompermaier]

Hi to all,

I'm trying to read a big file from my local fs, splitting it into two everything is fine, instead, reading it all at once I get this error>

java.io.IOException: Error of input/output

at java.io.FileInputStream.readBytes(Native Method)

at java.io.FileInputStream.read(FileInputStream.java:272)

at org.apache.flink.core.fs.local.LocalDataInputStream.read(LocalDataInputStream.java:76)

at org.apache.flink.api.common.io.DelimitedInputFormat.fillBuffer(DelimitedInputFormat.java:581)

at org.apache.flink.api.common.io.DelimitedInputFormat.readLine(DelimitedInputFormat.java:487)

at org.apache.flink.api.common.io.DelimitedInputFormat.nextRecord(DelimitedInputFormat.java:453)

at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:173)

at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:223)

at java.lang.Thread.run(Thread.java:745)

Do you know which could be the cause?

Best,

Flavio

[Flavio Pompermaier]

Probably this is a real disk error, I had an i/o error also copying that file in another location..I hope my disk is not dying..

On Sat, May 9, 2015 at 8:46 AM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm trying to read a big file from my local fs, splitting it into two everything is fine, instead, reading it all at once I get this error>

java.io.IOException: Error of input/output

at java.io.FileInputStream.readBytes(Native Method)

at java.io.FileInputStream.read(FileInputStream.java:272)

at org.apache.flink.core.fs.local.LocalDataInputStream.read(LocalDataInputStream.java:76)

at org.apache.flink.api.common.io.DelimitedInputFormat.fillBuffer(DelimitedInputFormat.java:581)

at org.apache.flink.api.common.io.DelimitedInputFormat.readLine(DelimitedInputFormat.java:487)

at org.apache.flink.api.common.io.DelimitedInputFormat.nextRecord(DelimitedInputFormat.java:453)

at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:173)

at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:223)

at java.lang.Thread.run(Thread.java:745)

Do you know which could be the cause?

Best,

Flavio

[rmetzger0]

The error is coming directly from the JVM. Reading that file without Flink would probably lead to the same error.

Maybe there is a hardware or operating system error.

On Sat, May 9, 2015 at 9:12 AM, Flavio Pompermaier <[hidden email]> wrote:

Probably this is a real disk error, I had an i/o error also copying that file in another location..I hope my disk is not dying..

On Sat, May 9, 2015 at 8:46 AM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm trying to read a big file from my local fs, splitting it into two everything is fine, instead, reading it all at once I get this error>

java.io.IOException: Error of input/output

at java.io.FileInputStream.readBytes(Native Method)

at java.io.FileInputStream.read(FileInputStream.java:272)

at org.apache.flink.core.fs.local.LocalDataInputStream.read(LocalDataInputStream.java:76)

at org.apache.flink.api.common.io.DelimitedInputFormat.fillBuffer(DelimitedInputFormat.java:581)

at org.apache.flink.api.common.io.DelimitedInputFormat.readLine(DelimitedInputFormat.java:487)

at org.apache.flink.api.common.io.DelimitedInputFormat.nextRecord(DelimitedInputFormat.java:453)

at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:173)

at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:223)

at java.lang.Thread.run(Thread.java:745)

Do you know which could be the cause?

Best,

Flavio