Re: Udf Performance and Object Creation

Posted by Stephan Ewen on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Udf-Performance-and-Object-Creation-tp2378p2387.html

Hi!

(1) A mapper is created once per parallel task. So if you create a program that runs a map() transformation with a parallelism of n, you will have n mapper instances in the cluster. Some may be on the same TaskManager, if the TaskManager has multiple slots.

(2) I would really like that. But it means Java has to deal with both managed and unmanaged memory at the same time, which is quite a heavy addition. C# has some form of support for that.

BTW: Where did you originally post these questions? I have not seen them before...

On Fri, Aug 14, 2015 at 5:43 PM, Flavio Pompermaier <[hidden email]> wrote:

Any insight about these 2 questions..?

On 12 Aug 2015 17:38, "Flavio Pompermaier" <[hidden email]> wrote:

This is something I've never understood in depth: isn't a mapper created for each record?if it's created only once per task manager then it's not so different from mapPartition..what I'm missing here?

And then a more philosophic question: all big data framework requires somehow to manage memory very efficiently (Flink has even though to reserve a fraction of the entire memory in order to have control over it). Wouldn't be simpler if java would finally release some APIs (even marked as unsafe, it doesn't change theMat much) to allow for a full control of the memory..?it will make a lot of sense for all big data platforms (at least for non-UDF code...).

Best,
Flavio

On 12 Aug 2015 12:44, "Timo Walther" <[hidden email]> wrote:

Hello Michael,

every time you code a Java program you should avoid object creation if you want an efficient program, because every created object needs to be garbage collected later (which slows down your program performance).
You can have small Pojos, just try to avoid the call "new" in your functions:

Instead of:

class Mapper implements MapFunction<String,Pojo> {
public Pojo map(String s) {
    Pojo p = new Pojo();
    p.f = s;
}
}

do:

class Mapper implements MapFunction<String,Pojo> {
private Pojo p = new Pojo();
public Pojo map(String s) {
    p.f = s;
}
}

Then an object is only created once per Mapper and not per record.

Hope this helps.

Regards,
Timo



On 12.08.2015 11:53, Michael Huelfenhaus wrote:

Hello

I have a question about the programming of user defined functions, is it still like in old Stratosphere times the case that object creation should be avoided al all cost? Because in some of the examples there are now Tuples and other objects created before returning them.

I gonna have an at least 6 step streaming plan and I am going to use Pojos. Is it performance wise a big improvement to define one big pojo that can be used by all the steps or better to have smaller ones to send less data but create more objects.

Thanks
Michael