Re: Avro Parquet/Flink/Beam

Posted by Jean-Baptiste Onofré on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Avro-Parquet-Flink-Beam-tp10590p10591.html

Hi,

Beam provides a AvroCoder/AvroIO that you can use, but not yet a
ParquetIO (I created a Jira about that and started to work on it).

You can use the Avro reader to populate the PCollection and then use a
custom DoFn to create the Parquet (waiting for the ParquetIO).

Regards
JB

On 12/12/2016 05:19 PM, Newport, Billy wrote:

> Are there any examples showing the use of beam with avro/parquet and a
> flink runner? I see an avro reader for beam, is it a matter of writing
> another one for avro-parquet or does this need to use the flink
> HadoopOutputFormat for example?
>
>
>
> Thanks
>
> Billy
>
>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com