http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Does-DataStream-writeAsCsv-suppose-to-work-like-this-tp3259.html
Hi, flinkers!
I'm new to this whole thing,
and it seems to me that ' org.apache.flink.streaming.api.datastream.DataStream.writeAsCsv(String, WriteMode, long)' does not work properly.
To be specific, data were not flushed by update frequency when write to HDFS.
what make it more disturbing is that, if I check the content with 'hdfs dfs -cat xxx', sometimes I got partial records.
I did a little digging in flink-0.9.1.
And it turns out all 'org.apache.flink.streaming.api.functions.sink.FileSinkFunction.invoke(IN)' does
is pushing data to 'org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream'
which is a delegate of 'org.apache.hadoop.fs.FSDataOutputStream'.
In this scenario, 'org.apache.hadoop.fs.FSDataOutputStream' is never flushed.
Which result in data being held in local buffer, and 'hdfs dfs -cat xxx' might return partial records.
Does 'DataStream.writeAsCsv' suppose to work like this? Or I messed up somewhere?
Best regards and thanks for your time!
Rex