Re: output writer

Posted by Michele Bertoni on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/output-writer-tp2296p2747.html

Hi guys,

sorry for late answer but I am still working to get this done but I don’t understand something

I do have my own writeRecord function, but that function is not able to open new output stream or anything else so I don’t understand how to do that

at first I think I should at least partition my data according to the output key (each key to one file)

then I need to name the file exactly with that key

but I don’t know how to go on

thanks

michele

Il giorno 30/lug/2015, alle ore 12:53, Radu Tudoran <[hidden email]> ha scritto:

Re-hi,

 

I have double –checked and actually there is an OutputFormat interface in flink which can be extended.

I believe that for this kind of specific formats as mentioned by Michele, each can develop the appropriate format.

On the other hand, having more outputformats I believe is something that could be contributed. We should identify a couple of common formats. The first one that comes in my mind is to have something for writing to memory (e.g. memory buffer)

 

 

 

Dr. Radu Tudoran

Research Engineer

IT R&D Division

 

<image001.png>

HUAWEI TECHNOLOGIES Duesseldorf GmbH

European Research Center

Riesstrasse 25, 80992 München

 

E-mail: [hidden email]

Mobile: [phone]

Telephone: [phone]

 

HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Jingwen TAO, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Jingwen TAO, Wanzhou MENG, Lifang CHEN

 

From: Fabian Hueske [[hidden email]] 
Sent: Thursday, July 30, 2015 11:34 AM
To: [hidden email]
Subject: Re: output writer

 

Hi Michele, hi Radu

Flink does not have such an OutputFormat, but I agree, it would be a valuable addition.

Radu's approach looks like the way to go to implement this feature.

@Radu, is there a way to contribute your OutputFormat to Flink?

Cheers, Fabian

 

2015-07-30 10:24 GMT+02:00 Radu Tudoran <[hidden email]>:

Hi,

My 2 cents ... based on something similar that I have tried.
I have created an own implementation for OutputFormat where you define your own logic for what happens in the "writerecord function". This logic would consist in making a distinction between the ids and write each to the appropriate file

Might be that other solutions exist


Dr. Radu Tudoran
Research Engineer
IT R&D Division


HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München

E-mail: [hidden email]
Mobile: <a href="tel:%2B49%2015209084330" style="color: purple; text-decoration: underline;" class="">[phone]
Telephone: <a href="tel:%2B49%20891588344173" style="color: purple; text-decoration: underline;" class="">[phone]

HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Jingwen TAO, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Jingwen TAO, Wanzhou MENG, Lifang CHEN

-----Original Message-----
From: Michele Bertoni [mailto:[hidden email]]
Sent: Thursday, July 30, 2015 10:15 AM
To: [hidden email]
Subject: output writer

Hi everybody,
I have a question about the writer
I have to save my dataset in different files according to a field of the tuples

let’s assume I have a groupId in the tuple, I need to store each group in a different file, with a custom name: any idea on how i can do that?


thanks!
Michele