Does Flink support TFRecordFileOutputFormat?

[殿李]

Hi,

Does Flink support TFRecordFileOutputFormat? I can't find the relevant information in the document.

As far as I know, spark is supportive.


Best regards
Peidian Li

[Danny Chan]

I didn’t see any class named TFRecordFileOutputFormat in Spark, for TF do you mean TensorFlow ?

Best,

Danny Chan

在 2020年7月10日 +0800 PM5:28，殿李 <[hidden email]>，写道：

Hi,

Does Flink support TFRecordFileOutputFormat? I can't find the relevant information in the document.

As far as I know, spark is supportive.


Best regards
Peidian Li

[殿李]

Hi,

Yes, TF means TensorFlow.

This class may not be in the spark package, but spark supports writing this file format to HDFS.

tfRDD.saveAsNewAPIHadoopFile(output, "org.tensorflow.hadoop.io.TFRecordFileOutputFormat",
                                keyClass="org.apache.hadoop.io.BytesWritable",
                                valueClass="org.apache.hadoop.io.NullWritable”)

> 2020年7月13日 下午2:21，Danny Chan <[hidden email]> 写道：
>
> I didn’t see any class named TFRecordFileOutputFormat in Spark, for TF do you mean TensorFlow ?
>
> Best,
> Danny Chan
> 在 2020年7月10日 +0800 PM5:28，殿李 <[hidden email]>，写道：
>> Hi,
>>
>> Does Flink support TFRecordFileOutputFormat? I can't find the relevant information in the document.
>>
>> As far as I know, spark is supportive.
>>
>>
>> Best regards
>> Peidian Li

[Jingsong Li]

Hi,

Flink also has `HadoopOutputFormat`, it can wrap hadoop OutputFormat to Flink sink.

You can have a try.

Best,

Jingsong

On Mon, Jul 13, 2020 at 2:34 PM 殿李 <[hidden email]> wrote:

Hi,

Yes, TF means TensorFlow.

This class may not be in the spark package, but spark supports writing this file format to HDFS.

tfRDD.saveAsNewAPIHadoopFile(output, "org.tensorflow.hadoop.io.TFRecordFileOutputFormat",
                                keyClass="org.apache.hadoop.io.BytesWritable",
                                valueClass="org.apache.hadoop.io.NullWritable”)



> 2020年7月13日 下午2:21，Danny Chan <[hidden email]> 写道：
>
> I didn’t see any class named TFRecordFileOutputFormat in Spark, for TF do you mean TensorFlow ?
>
> Best,
> Danny Chan
> 在 2020年7月10日 +0800 PM5:28，殿李 <[hidden email]>，写道：
>> Hi,
>>
>> Does Flink support TFRecordFileOutputFormat? I can't find the relevant information in the document.
>>
>> As far as I know, spark is supportive.
>>
>>
>> Best regards
>> Peidian Li

--

[殿李]

Thanks, I'll check it out. 

Jingsong Li <[hidden email]> 于2020年7月13日周一 下午2:50写道：

Hi,

Flink also has `HadoopOutputFormat`, it can wrap hadoop OutputFormat to Flink sink.

You can have a try.

Best,

Jingsong

On Mon, Jul 13, 2020 at 2:34 PM 殿李 <[hidden email]> wrote:

Hi,

Yes, TF means TensorFlow.

This class may not be in the spark package, but spark supports writing this file format to HDFS.

tfRDD.saveAsNewAPIHadoopFile(output, "org.tensorflow.hadoop.io.TFRecordFileOutputFormat",
                                keyClass="org.apache.hadoop.io.BytesWritable",
                                valueClass="org.apache.hadoop.io.NullWritable”)



> 2020年7月13日 下午2:21，Danny Chan <[hidden email]> 写道：
>
> I didn’t see any class named TFRecordFileOutputFormat in Spark, for TF do you mean TensorFlow ?
>
> Best,
> Danny Chan
> 在 2020年7月10日 +0800 PM5:28，殿李 <[hidden email]>，写道：
>> Hi,
>>
>> Does Flink support TFRecordFileOutputFormat? I can't find the relevant information in the document.
>>
>> As far as I know, spark is supportive.
>>
>>
>> Best regards
>> Peidian Li

--

Best, Jingsong Lee

--