Compression Streamingfilesink ROW-encoded format

[Vijayendra Yadav]

Hi Team,

Is there a way to enable compression in StreamingFileSink API for Row-encoded formats ?.

val sink: StreamingFileSink[String] = StreamingFileSink
    .forRowFormat(new Path(outputPath), new SimpleStringEncoder[String]("UTF-8"))

Regards,

Vijay

[Ravi Bhushan Ratnakar]

Hi Vijayendra,

You could achieve row encoded with like this as well

codecName = "org.apache.hadoop.io.compress.GzipCodec"

val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()

Regards,

Ravi

On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <[hidden email]> wrote:

Hi Team,

Is there a way to enable compression in StreamingFileSink API for Row-encoded formats ?.

val sink: StreamingFileSink[String] = StreamingFileSink
    .forRowFormat(new Path(outputPath), new SimpleStringEncoder[String]("UTF-8"))

Regards,

Vijay

[Vijayendra Yadav]

Hi Ravi,

Thanks for your response. But your example is for forBulkFormat. How about  forRowFormat ?.

Regards,

Vijay

On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <[hidden email]> wrote:

Hi Vijayendra,

You could achieve row encoded with like this as well

codecName = "org.apache.hadoop.io.compress.GzipCodec"

val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()

Regards,

Ravi

On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <[hidden email]> wrote:

Hi Team,

Is there a way to enable compression in StreamingFileSink API for Row-encoded formats ?.

val sink: StreamingFileSink[String] = StreamingFileSink
    .forRowFormat(new Path(outputPath), new SimpleStringEncoder[String]("UTF-8"))

Regards,

Vijay

[Ravi Bhushan Ratnakar]

Hi Vijayendra,

As far as rowFormat is concerned, it doesn't support compression. 

Regards, 

Ravi 

On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav, <[hidden email]> wrote:

Hi Ravi,

Thanks for your response. But your example is for forBulkFormat. How about  forRowFormat ?.

Regards,

Vijay

On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <[hidden email]> wrote:

Hi Vijayendra,

You could achieve row encoded with like this as well

codecName = "org.apache.hadoop.io.compress.GzipCodec"

val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()

Regards,

Ravi

On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <[hidden email]> wrote:

Hi Team,

Is there a way to enable compression in StreamingFileSink API for Row-encoded formats ?.

val sink: StreamingFileSink[String] = StreamingFileSink
    .forRowFormat(new Path(outputPath), new SimpleStringEncoder[String]("UTF-8"))

Regards,

Vijay

[Vijayendra Yadav]

Thank You Ravi for Quick help. One Last Question is this compression supported with Flink Version 1.10.0 ?

Regards,

Vijay

On Tue, Jul 28, 2020 at 1:20 PM Ravi Bhushan Ratnakar <[hidden email]> wrote:

Hi Vijayendra,

As far as rowFormat is concerned, it doesn't support compression. 

Regards, 

Ravi 

On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav, <[hidden email]> wrote:

Hi Ravi,

Thanks for your response. But your example is for forBulkFormat. How about  forRowFormat ?.

Regards,

Vijay

On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <[hidden email]> wrote:

Hi Vijayendra,

You could achieve row encoded with like this as well

codecName = "org.apache.hadoop.io.compress.GzipCodec"

val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()

Regards,

Ravi

On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <[hidden email]> wrote:

Hi Team,

Is there a way to enable compression in StreamingFileSink API for Row-encoded formats ?.

val sink: StreamingFileSink[String] = StreamingFileSink
    .forRowFormat(new Path(outputPath), new SimpleStringEncoder[String]("UTF-8"))

Regards,

Vijay

[Ravi Bhushan Ratnakar]

Yes, flink-compress module is  supported from  1.10.0 and onward. 

Regards, 

Ravi 

On Tue 28 Jul, 2020, 23:11 Vijayendra Yadav, <[hidden email]> wrote:

Thank You Ravi for Quick help. One Last Question is this compression supported with Flink Version 1.10.0 ?

Regards,

Vijay

On Tue, Jul 28, 2020 at 1:20 PM Ravi Bhushan Ratnakar <[hidden email]> wrote:

Hi Vijayendra,

As far as rowFormat is concerned, it doesn't support compression. 

Regards, 

Ravi 

On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav, <[hidden email]> wrote:

Hi Ravi,

Thanks for your response. But your example is for forBulkFormat. How about  forRowFormat ?.

Regards,

Vijay

On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <[hidden email]> wrote:

Hi Vijayendra,

You could achieve row encoded with like this as well

codecName = "org.apache.hadoop.io.compress.GzipCodec"

val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()

Regards,

Ravi

On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <[hidden email]> wrote:

Hi Team,

Is there a way to enable compression in StreamingFileSink API for Row-encoded formats ?.

val sink: StreamingFileSink[String] = StreamingFileSink
    .forRowFormat(new Path(outputPath), new SimpleStringEncoder[String]("UTF-8"))

Regards,

Vijay

[Vijayendra Yadav]

Thank You.

On Wed, Jul 29, 2020 at 2:07 AM Ravi Bhushan Ratnakar <[hidden email]> wrote:

Yes, flink-compress module is  supported from  1.10.0 and onward. 

Regards, 

Ravi 

On Tue 28 Jul, 2020, 23:11 Vijayendra Yadav, <[hidden email]> wrote:

Thank You Ravi for Quick help. One Last Question is this compression supported with Flink Version 1.10.0 ?

Regards,

Vijay

On Tue, Jul 28, 2020 at 1:20 PM Ravi Bhushan Ratnakar <[hidden email]> wrote:

Hi Vijayendra,

As far as rowFormat is concerned, it doesn't support compression. 

Regards, 

Ravi 

On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav, <[hidden email]> wrote:

Hi Ravi,

Thanks for your response. But your example is for forBulkFormat. How about  forRowFormat ?.

Regards,

Vijay

On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <[hidden email]> wrote:

Hi Vijayendra,

You could achieve row encoded with like this as well

codecName = "org.apache.hadoop.io.compress.GzipCodec"

val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()

Regards,

Ravi

On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <[hidden email]> wrote:

Hi Team,

Is there a way to enable compression in StreamingFileSink API for Row-encoded formats ?.

val sink: StreamingFileSink[String] = StreamingFileSink
    .forRowFormat(new Path(outputPath), new SimpleStringEncoder[String]("UTF-8"))

Regards,

Vijay