Re: Reading Data from zip/gzip

Posted by Amit Jain on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Reading-Data-from-zip-gzip-tp23991p23993.html

Hi Chris,

FileInputFormat automatically takes cares of file decompression for the files with gzip, xz, bz2 and deflate extensions. 

--

Thanks,

Amit

Source: https://github.com/apache/flink/blob/7b040b915504e59243c642b1f4a84c956d96d134/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java#L118

private static void initDefaultInflaterInputStreamFactories() {
   InflaterInputStreamFactory<?>[] defaultFactories = {
         DeflateInflaterInputStreamFactory.getInstance(),
         GzipInflaterInputStreamFactory.getInstance(),
         Bzip2InputStreamFactory.getInstance(),
         XZInputStreamFactory.getInstance(),
   };
   for (InflaterInputStreamFactory<?> inputStreamFactory : defaultFactories) {
      for (String fileExtension : inputStreamFactory.getCommonFileExtensions()) {
         registerInflaterInputStreamFactory(fileExtension, inputStreamFactory);
      }
   }
}

On Mon, Oct 22, 2018 at 2:03 PM chrisr123 <[hidden email]> wrote:

I'm able to read normal txt or csv files using Flink,
but what would I need to do in order to read them if they
are given to me in zip or gzip format? Assuming I do not want
to have to unzip them.
Thanks!




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/