override file in flink

[subashbasnet]

Hello all,

I am trying to write the streaming data to file and update it recurrently with the streaming data. I get the following unable to override exception error:

Caused by: java.io.IOException: File or directory already exists. Existing files and directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and directories.

at org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:594)

at org.apache.flink.api.common.io.FileOutputFormat.open(FileOutputFormat.java:227)

at org.apache.flink.api.java.io.CsvOutputFormat.open(CsvOutputFormat.java:160)

at org.apache.flink.streaming.api.functions.sink.OutputFormatSinkFunction.open(OutputFormatSinkFunction.java:60)

at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)

at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)

at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:314)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:214)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)

at java.lang.Thread.run(Thread.java:745)



How to set the necessary override property?

Best Regards,
Subash Basnet

[Chiwan Park-2]

Hi subash,

You can pass WriteMode in second parameter of write* method. For example:

```
DataStream<…> myStream = …;
myStream.writeAsCsv(“path of output”, FileSystem.WriteMode.OVERWRITE);
```

I hope this helps.

Regards,
Chiwan Park

> On Mar 22, 2016, at 8:18 PM, subash basnet <[hidden email]> wrote:
>
> Hello all,
>
> I am trying to write the streaming data to file and update it recurrently with the streaming data. I get the following unable to override exception error:
>
> Caused by: java.io.IOException: File or directory already exists. Existing files and directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and directories.
> at org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:594)
> at org.apache.flink.api.common.io.FileOutputFormat.open(FileOutputFormat.java:227)
> at org.apache.flink.api.java.io.CsvOutputFormat.open(CsvOutputFormat.java:160)
> at org.apache.flink.streaming.api.functions.sink.OutputFormatSinkFunction.open(OutputFormatSinkFunction.java:60)
> at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)
> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:314)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:214)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> How to set the necessary override property?
>
> Best Regards,
> Subash Basnet
>

[Chesnay Schepler]

by using DataStream#writeAsCsv(String path, WriteMode writeMode)

On 22.03.2016 12:18, subash basnet wrote:

Hello all,

I am trying to write the streaming data to file and update it recurrently with the streaming data. I get the following unable to override exception error:

Caused by: java.io.IOException: File or directory already exists. Existing files and directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and directories.

at org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:594)

at org.apache.flink.api.common.io.FileOutputFormat.open(FileOutputFormat.java:227)

at org.apache.flink.api.java.io.CsvOutputFormat.open(CsvOutputFormat.java:160)

at org.apache.flink.streaming.api.functions.sink.OutputFormatSinkFunction.open(OutputFormatSinkFunction.java:60)

at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)

at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)

at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:314)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:214)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)

at java.lang.Thread.run(Thread.java:745)



How to set the necessary override property?

Best Regards,
Subash Basnet